Understanding how to find and parse data: Part 1

So in the past few days I have been asked by different people that I am aquatinted with about getting a hold of data from several sources and aggregating it into a central location. Now I have a varied background in development and IT so this is really something that should be covered in a case by case basis but I am going to craft a use case that I think will get the concepts across so you can start you work on your specific problem.

Data Mining, enter the E.T.L.

I spent a few years working with a very,  very large telecommunications company (you can probably guess who) and they collect a shit load of data. Network data, phone call data, internet activity, website activity, and many more things. Collecting this data can be a hell of a task, data formats are all over the place along with different connection protocols. This is where an E.T.L. Comes into play. It stands for Extract, Translate, and Load. You extract data from a target, translate it into a format that suits your need, and load it into your own database. This sounds simple in practice and in some cases it is. There are other times that this becomes a work of art as you have to learn how to reverse engineer a system just to get the data you need.

A Simple Use Case

Realtor.com is a great example of a data rich target along with a structure that you have to navigate in order to get exactly what you are looking for. So lets start with a simple search: http://www.realtor.com/realestateagents/08052 this will give you all the agents that are in the area of the 08052 zip code.

This is a clear, repeatable display. Now lets take a quick peak at the code behind it. I use a tool called firebug, its great if you do any web development (http://getfirebug.com/)

This will give you a box like this:

Now if you look, you can see <div class=”agent-list-card clearfix” data-url=”http://www.realtor.com/realestateagents/Keri–Ricci_CHERRY-HILL_NJ_4532_179994593″>. Now look at the pattern: class=”agent-list-card clearfix” is used for each of the agents in what they are calling “cards”. Also the data-url has the link to the agents direct page.

Now, with a little ingenuity you can build a little database…

Enter the Dragon (or Snake in this case)

Python is an extremely versatile and easy to use programming language. I am not going to cover Python tutorials here but you can find a good one here: https://docs.python.org/2/tutorial/

 

To be continued…

What I am going to show you in part 2 is how to connect to a site and work your way to building a comma separated list that we can eventually turn into a database.

NSNG – 30 day plan failure and what I plan to do about it

OK so technically right now I’m on day 20. On or about day 15, I have a little sugar in the form of a peanut butter cup. Then on day 16 a few m&ms at work and then more on cups and I have been slowly falling off the rails and now a full serving of ice cream… I can’t stop, I can walk away from bread and grains without a 2nd thought but take away my Hershey chocolate and I start twitching like a fiend.

I’m going to see what I can find out about my sugar addiction and what to do about it. I will post my findings here… 

Once I have a plan I will restart my 30 day plan and try and complete it this time.

Cooking with Cast Iron

So cooking with cast iron… it can be scary if you have never used it before but if you take care of your pans they will love you back. In fact you can hand them down to your children and even grandchildren if you teach them the right way to care for them.

Seasoning

A new cast iron pan will need a quick rinse with water, but never EVER use soap! I like to use olive oil for almost everything but you could also use canola or even grape seed oil. cover the whole thing, use a thin cloth to spread it.

 

Once it’s covered, go a head and cook on it! Make a nice steak or a chicken. Once you are done clean it immediately while it’s still hot. Again use some water. You can use a plastic bristle brush if you need but again no soap!

Once it’s clean, dry it off and cover it in oil again, you will leave it like this until the next time you cook on it. The coating of oil will keep it from rusting. If you have a hard to clean spot do not use a wire brush or SOS…

Take some oil and a tablespoon of salt and pile the salt over the crud. Now time for some elbow grease… Get a paper towel and fold it up a few time and use that on the salt, grind it into the crud until it’s gone. Once you are done, rinse it off, dry it, and oil it again.

This is a tedious process but trust me it’s worth it.

General Use

Cast iron heats evenly all the way into the handle. Use a towel to use your pan so you don’t burn your self. Use wooden or plastic/silicon utensils. Try not to use metal if you can help it.

Cast iron can also be put in the oven. It’s great for making pies and cobblers in. Also, because they heat so evenly, you can fill it half way up with oil and fry away 🙂

If all else fails, you can always use it as a weapon 🙂

BBQ Links and Resources

So every once and a while I will send out emails to the people on my list and I ask a short 5 question survey so I can keep track of what you guys are really looking for content wise. This past survey brought up some things that I wanted to answer right away.

I would like info on where maybe bbq classes are being offered

This is really dependent on where you live in the world. This list might help, some have their own class page others mix them in with the events pages.

USA

Canada

Australia

Affiliates:

If this conversation is getting you hungry, try out The Pig Of The Month Club They have Ribs, Pork, Sausage, Chicken, and many other meats all sent directly to you once each month! They also carry Rubs and Sauces for your enjoyment as well.

UK

External Resources

There are a few places other than my podcast that are giving out great info. Here is a list of resources that you might like.