Data-Analysis on Junjay Tan

Best Python Data Cleaning Libraries For People Data

Tue, 01 Sep 2015 00:00:00 +0000

In the past year I’ve had to do various projects that needed data clean up of typical contact info fields like emails and phone numbers. You’d think people would have solved these problems by now, and that have (kind of). If you use Python to do data cleaning, here are some libraries you will find useful:

Validating Emails Link to heading

Josh Tauberer, a known developer in the open government data movement, has developed an email validation library.

How to Calculate Income by Zip Code From US Census Data

Tue, 07 Aug 2012 00:00:00 +0000

In this post I’ll explain how I tabulated mean income by household from the 2010 American Community Survey by zip code.

The US Census provides lots of data at various geographical scales, but you often have to dig through its web site and combine tables together to get what you want. Unfortunately income by zip code is one of these data sets.

The good news is you can get a proxy for household income, household median income, and per capita income by zip code . The bad news is that unless you want to pay for a data source from ESRI or some other vendor, you’ll have to do some legwork to get this data.

Linear Interpolation in MySQL

Sun, 03 Jun 2012 00:00:00 +0000

ySQL is often used to store time-series data. But it doesn’t offer a way to interpolate between time points which is useful in a lot of data analysis work.

(Interpolation is used to estimate values for periods between where you have actual measurements, and allows you to look at your data in a more continuous fashion representative of how things occur in the real world).

This can be a problem if you have granular time series data and want to analyze them at an aggregated level.