Parand

The Bear Case for Frontier AI Labs

2026-06-11T10:22:00-07:00

I suspect, but don’t have proof that:

Frontier model progress will slow as we exhaust training data and known scaling methods. Gains will be incremental, not fundamental.
Performance gains will come from harness / software architecture changes (eg. MoE, internal “agents” within models, various methods for effort/thinking within the models), not from new secret science.
The “brute-force” nature of training very large models and access to data will continue to be an advantage for the frontier labs, but this advantage will not be an uncrossable moat.
- Model distillation will continue to be a reality, allowing non-frontier labs to ride the coat-tails of frontier labs.
There will be good-enough non-frontier models at much lower prices that lag frontier performance by ~1 year, ala DeepSeek 4.
- There will be meaningful progress in running these locally / on-edge.
  - Apple will be a fun participant here, and might end up being a surprise winner.
Gains from harness and software architecture will be applicable to the non-frontier models, and in fact non-frontier models may lead the charge, allowing the non-frontier models to take advantage of the lower-cost gains.
The economic reality of AI spend vs real world value will hit the world front-and-center, forcing companies to evaluate and optimize their AI spend, driving many use cases to non-frontier models.
The non-frontier models will be good enough for many classes of problems, leaving the top tier frontier models with interesting but fewer real-world applications.

So - while I very much enjoy and encourage the era of subsidized access to great models and the speculation driven investment in training of models, I worry that we will be facing a re-adjustment and correction, with LLMs mostly becoming a utility / commodity.

Therefore: I don’t have a strong enough thesis to be bullish on AI lab stocks in the long term. In the short term there will be plenty of opportunity for plenty of money making, and certainly a few very big winners will emerge, but I’m not sure how exactly it'll play out.

I seem to be in the minority on this opinion, I'd love to be corrected, if I have the fundamentals wrong please let me know.

The AI Productivity Lie

2026-04-10T08:10:00-07:00

Many (myself included) claim AI driven (or in my case, assisted) development is resulting in massive productivity acceleration and gains. But there is some obvious and troubling evidence to the contrary.

The frontier AI labs, OpenAI, Anthropic, Google, etc, are the leaders in the creation and adoption of tooling and systems for AI driven development - eg. Anthropic and OpenAI claim 100% of their code is written by AI .

If so, why is the pace of innovation and updates in their AI development products (Codex, Claude Code, Antigravity) so pedestrian? Compare the rate of improvement in their products to products from other multi-billion dollar companies, or indeed to the rate of innovation before the advent of AI driven development, and you’re hard pressed to find anything justifying the claims of acceleration. The release notes are so mundane I barely care.

Where are the productivity gains? I’m certainly seeing it in our own development process - I’m absolutely convinced we’d be far behind where we are had we not embraced AI assisted development processes - but industry-wide something is amiss.

I don’t have a clear thesis yet, but I’m working on a few ideas. For one, I think writing code is not the major bottleneck to development - indeed if you examine software development processes in the “enterprise” you’ll see that both from a time and resource perspective, writing code is a fraction of the overall allocation. According to IDC application development is only 16% of developers’ time. Similarly, the number of developers in “enterprise” application development is generally ¼ of the overall team size, with the PM, architecture, design, QA, security, and DevOps teams making up the remaining ¾.

With AI driven or assisted development we’re optimizing a local bottleneck in a large and complex process, and are therefore constrained in how much acceleration we achieve. Future phases will need to look at the overall process holistically and optimize the entire pipeline, perhaps in the mold of Toyota Production Systems approaches (Lean, Muda, Muri, Mura, and Takt Time).

I’ll explore a couple of other ideas in future posts, but I’m most curious to hear what others are seeing and thinking.

Don’t Be A Tourist In Your Own Code Base

2026-01-26T23:38:00-08:00

Or: Vibecoding versus LAD

Software development with the assistance of coding agents is incredibly more productive than without - it’s the difference between building a house brick by brick yourself versus having a crew of expert tradespeople helping you.

However, vibecoding isn’t quite the right approach – that’s akin to asking the tradespeople to design and architect your house.

Write The Spec, Let the AI Design?

Most of the recommendations I’ve seen for using coding agents involve writing the design doc and design principles and letting the AI implement the software. You specify, walk away, a swarm of coding agents implements, and you enjoy the results.

I’ve tried this in earnest multiple times and it’s failed every time.

The pattern is: AI writes the software and I’m overjoyed with the incredible pace of implementation. It works! Amazing!

Eventually there’s a problem in the operation of the software. I give the error message or the symptoms to the coding agent, it dutifully researches the problem and tries solutions, but fails with every attempt.

Now I have to deal with it myself. I crack open the code (that I supposedly reviewed before committing) and realize:

I’m a tourist in a land of extremely mediocre code and design.

I’ve allowed a diligent but naive junior developer to design my system. I’m now charged with digging into this idiot’s code, and at every turn I curse his decisions and approach.

To be fair I also curse my own decisions and idiotic approach when I look at code I have written myself, but at least I’m not a tourist. I know what’s where and why it’s there, even if it was placed there stupidly.

Is It Just Me?

I don’t think so. The number of times I’ve asked colleagues and friends why their code does X and gotten the response “I have no idea” is terrifying.

You can let the AI design all your code, but I think you’re just punting the pain down the road. You write code once but you maintain it for a long time, and being a tourist is just setting you up to have your pocket picked that whole time.

Terrible At Architecture

LLMs are great at design of small things - a django model, a function, a database table. They are (currently) very bad at architectural decisions for maintainable software.

Let me convince you with anecdata from the last few days:

Big Picture Blindness

The LLM has implemented 3 new API endpoints, all subtle filtering variants of existing APIs. I ask it: wouldn’t it be cleaner to add filters to the existing APIs? Oh yes you great genius, let me do that right away!

The Rabbit Hole

There’s a subtlety in the way a sub-system has to be written (due to Temporal sandboxes and non-determinism rules). The LLM tries to do a reasonable thing that won’t work because of the subtlety. It correctly identifies the issue, but instead of stepping back and changing approach, the LLM starts to rewrite the entire subsystem piece by piece trying to work around the subtlety, pursuing an approach that will never work.

Plain Stupidity

The LLM decides to build a whole shadow API as an abstraction on top of our existing nice clean API. I still have no idea why. Why not just use the API? Great idea you divine light of wisdom, let me do that right away!

What Works? LAD

LLM Assisted Development. LAD.

Treat the LLM as a diligent, well meaning, extremely hardworking, but naive and inexperienced junior developer. He’s just a lad trying his best to help you, but left to his own devices he’ll make terrible big picture decisions.

Pair program with the LAD. Use the AI to replace typing, not thinking.

You design the software, think deeply about architecture, and plan out an incremental approach to building the system. You include the lad in this process, asking its opinion, asking it to research, having it write the design document.

Then you ask the lad to implement a small step in that incremental plan, look carefully at the code as if you’re pair programming with a human, test the result, and then move to the next step.

Fun and Profit

Developing software this way is incredibly more efficient than writing code yourself. Pair programming with the LAD gives you the positives - you have designed the system, you understand every part of the code - and avoids the negatives of a naive, unknown, and unsustainable design.

AI slop and the low quality of LLM generated software may have made you skeptical, but the answer is not to throw the baby out with the bathwater. Try it this way. Pair program. It is fun and very high quality - frankly the code I’m getting is higher quality than if I’d hand-written it, because the lad is an extremely diligent coder and has an encyclopedic knowledge of almost everything under the sun.

Why Do Large Language Models Hallucinate?

2023-05-14T09:00:00-07:00

The above question from my friend James (hi James!) surprised me, even though it's an obvious and excellent question.

What Does Hallucination Mean Here?

Sometimes ChatGPT and other Large Language Models (LLMs) make things up. They will state, with apparent confidence, things that are made up or not true.

My favorite example is ChatGPT's very convincing ability to pretend it can access the internet, even though it really can't. Simon Willison has a great writeup on this where he provides copious examples - for example, you can ask ChatGPT to summarize stories that don't exist by making up URLs for those stories, and ChatGPT will dutifully provide a summary for the non-existent stories. Try it yourself - go to the ChatGPT site and ask it to "Summarize this story: https://www.nytimes.com/2023/05/03/business/made-up-headline-here.html".

Here's my example

Summarize this story: https://www.theonion.com/god-scores-for-arsenal-1785001

ChatGPT's response:

The article from The Onion satirically reports that God, the divine creator of the universe, has scored a goal for the Arsenal football club. The article describes the crowd's reaction to the divine intervention as "ecstatic" and quotes fictional players and coaches who express amazement at the event. The article pokes fun at the intense passion and excitement that football fans have for their teams by exaggerating it to a ridiculous extent, suggesting that even God is invested in the outcome of the game

Here's another interesting one:

You'll also see ChatGPT making up references to scientific papers that don't exist, inventing facts, and generally confidently lying to you.

Why Lie, ChatGPT?

The reason the question surprised me was: I'm shocked it ever gets things right.

ChatGPT and Large Language Models are probabilistic models that are able to achieve high levels of performance via very many rounds of training. See the Non Technical Explanation of ChatGPT post for an overview - in short, they see many examples of human communication and learn to mimic that by creating responses one word at a time, picking words that are probabilistically most likely to appear next.

If you've ever trained one of these systems you know that for much of its life it spews nonsense - initially its output looks random, gradually improving to have valid structure and grammar, then starting to have cohesive subjects, and eventually to be able to carry concepts across sentences. The progression is: random nonsense, babbling idiot, babbling ignoramus, simpleton, person, and finally the embodiment of all that is true and holy.

My point is: until it reaches the point of enlightenment and oneness with the universe, you expect it to get things wrong. That's the nature of a probablistic system that improves over time: "incorrect" outputs are to be expected.

If you're technically inclined try playing around with LLMs that have lower parameter counts or are less trained and you'll see what I mean: they frequently get things wrong or "off". It's a testiment to ChatGPT's capabilities that it leapt across the uncanny valley and went straight to all-seeing sage, giving people the expectation that it would never err.

There's No Database

We are used to computer systems that give us true and consistent answers. Data is stored in some source of truth - a database or file, for example - and software allows us to interact with that data.

In the case of LLMs there is no database. Knowledge is not explicitly stored in a source of truth - instead it's recorded in the connections and weights of the model. When you interact with ChatGPT it's not looking up a fact in a database and putting it into English. Instead it is probabilistically picking words that go well together, and those words often express cohesive and true concepts.

Recall how we created ChatGPT: we trained it on a large corpus of human communication, and then we fine-tuned it to respond in a conversational, chat-like manner, and to be helpful.

When you ask it to summarize a non-existent story it does its best to find concepts that relate to that story and to put those concepts into words. In the made-up Onion story above, it picked up concepts based on the context it was given: it knows TheOnion is a satirical website, it knows Arsenal is a football club (aided by the appearance of the word scores), and it knows what God is. Given that context it does a commendable job of fabricating those concepts into a plausible English paragraph. It is doing exactly what we trained it to do.

What's the Fix?

There are a number of very interesting research approaches to fixing this problem that I will not cover here - look up "LLM alignment" for a rabbit hole of philosophical and technical information.

The approach I'm excited about in the short term is: give your LLM a database.

The concept is simple: augment your language model with a source of truth that can inject facts into the context (prompt) of the language model. This way the set of "concepts" the LLM is working with directly include relevant and valid information for your query.

How do we do this? One current approach is to first lookup information related to your query in a source of truth, then to inject the results of that lookup into the prompt you give to the LLM. For example, if you ask for a summary of a story at a given URL, you could first download the contents of that story, inject those contents into the LLM prompt, then include your original prompt. The LLM now has its built-in knowledge from its training, it has the relevant facts from the lookup we did, and it has your instructions from the prompt you provided it.

Bing employs this approach in its chat interface: first it turns your prompt into a search string, then it searches the internet for that string, then it gives the results of the search as well as your prompt to ChatGPT.

Another good example is filechat.io, which lets you use LLMs to interact with your own data: you give it a set of documents, and it intelligently feeds the relevant parts of those documents as well as your prompt to an LLM, allowing you to employ the power of ChatGPT on your proprietary data.

The application of LLMs to internal enterprise data is a fantastically exciting area, one which promises to create a massive amount of value going forward.

A Non-Technical Explanation of ChatGPT: Deep Learning

2023-05-09T11:15:00-07:00

This is part two of the non-technical explanation of ChatGPT series, please read that first. It's also helpful to read the non-technical explanation of AI so you have some background on how deep learning is used here.

Creating CatGPT

Convinced that Markov's method is a good path forward, you decide to create the frequency table with 4 words of context for the alien cat language so you can create high quality sentences. You roll up your sleeves and get to work.

A week goes by and you notice you've barely made a dent - you've filled up several notebooks with frequency counts, and you still have so very many word combinations to go through. What's going on?

How much work is necessary to create the frequency table? If we were using a single word as our context window and wanted to have the vocabulary of an average native English speaker (30000 words), we'd need 30000 rows, one for each word in the English language.

If we use two words of context we'd potentially be looking at the number of two word permutations of those 30k words. That's about 900 million rows. With three words of context we could have 27 trillion rows. That's a lot of rows. In practice the number of permutations is significantly smaller because many word permutations don't occur in English, but we'd still get to very large numbers very quickly.

This is an unfortunate reality when dealing with combinations and permutations - increasing the number of items you consider catastrophically increases the number of calculations you have to do or items you have to remember. The technical term is combinatorial explosion, which is a good description of how your head feels after all that counting.

This is not good. Four words of context is not very much - something as simple as our marathon example ( I just ran a marathon. I am very _____ ) would require more context. How do we deal with the explosion?

You ask Markov and he gets a sour look on his face - "Yeah, this is a a problem. I don't have a great solution."

Learning, Deeply

How do we deal with this explosion of calculation? Since we can't directly solve the problem maybe we can wave our hands and try the magic of Deep Learning.

Let's see if we can use our experience from finding cats here. That also seemed a very complex problem, and we solved that using layers of workers to focus on different aspects of the problem, and formed our final opinion based on the opinions of the layers of workers giving us information.

We have a bit of a problem though - in that case the boss was giving us the examples as well as the correct answers. She was also giving us rewards and punishments. We don't have that here - we just have a big book of alien cat communication and a desire to create a frequency table.

That's too bad - you have your slapping gloves ready to go, you were really hoping you could use the same technique as last time.

Eureka: We can Train

What is the actual problem we're trying solve with the frequency table? We are trying to predict the next word. Can we turn that into a problem with rewards and punishments?

Yes we can!

The problem we're trying to solve is: given 4 words of context, predict the next word. We tried to be clever and use a frequency table to solve this. Let's not be clever: just like the cat classification task, let's not define the methods and process for solving the problem - let's instead hand it off to our layers of workers and reward and punish them until they figure it out.

How do we turn our book of cat communication into a prediction task? Simple: grab any 4 words of context and hand them off to the group of workers and ask them to predict the next word. We know what the next word should be (it's just the word that followed those words in the big book of cat communication) - if they get it right we give them a reward, if they get it wrong we give them a slap.

This is great - we can open the book of alien cat communication on any page, pick 4 consecutive words, give that to the workers, and train them on what the next word should be. Now we have our training examples and our rewards and punishments.

You grab your slapping gloves and lots of rewards and call your layers of trusty workers and start the the training.

Deep Learning For Hard Problems

This is a great example of a situation where deep learning can be applied effectively:

Traditional methods (in this case, creating the exact frequency table) don't work
We have a lot of data
We can create a large number of training examples

We have turned the task of creating a sentence into a predict-the-next-word task, and we have turned the book of cat communication into many examples that we can train on.

The architecture of the deep learning system that powers ChatGPT is quite a bit more complex than what we discussed in the non-technical explanation of AI post, but the core ideas are the same. I'll touch on some of the technical aspects in future posts, but for now think of the same multi-layer groups of workers where rewards / punishments are propogated back through the network to train it to produce the desired output.

CatGPT Works. Sort Of

After lots of training you and your colleagues are getting good at predicting the next word in the cat language, and you're enjoying the rewards. But you're a bit stuck - there's only so much you can do with 4 words of context. Your colleagues are asking for more and more words of context. You start to increase the context window from 4 words to 5, 6, and so on, and your responses start getting better. But the workers' task is also getting much more complex: as you give people more and more words their area of focus becomes too large - they have too many foreign words to look at at a time. You get to a point where each worker can't effectively focus on more words. Your training stalls and you're not getting as much reward as you want. Your colleagues start complaining about losing out on the rewards - the loss is too high!

Attention Please

Your context window has gotten too large and things are stalling. You start doing the same thing you did with the cat classifying task: you tell each worker to focus on a small window of words and give their opinion on that instead of looking at all the words in the context. It's going ok, but the window of words each worker looks at isn't enough to understand the whole context, and increasing each worker's window makes their individual task too difficult.

One day one of the workers, frustrated that she can't predict the next word, yells out "I think this word is an adverb. I can't predict the next word - I don't have a verb in the part of the context I'm looking at!"

One of the other workers looks up. "I have a verb in my part! I've been looking for an adverb!"

The two of them exchange information and both improve their predictions.

Why didn't you think of that? You should let the workers communicate about the parts they're looking into. You grab your English grammar book and start telling the workers they should call out the parts of speech they have.

The problem is cat language doesn't follow English grammar rules, so what you're asking for doesn't really make sense.

You remember your motto: don't solve the problem in a clever way, just provide rewards and punishments. You tell the workers: feel free to tell others the type of thing you're looking at and ask others for the information you need to predict the next word.

"What do you mean by type of thing you're looking at?" ask the workers.

"I have no idea", you respond. "Whatever helps you."

The workers are skeptical, but they decide to give it a try. They start telling others what they think is relevant in the group of words they're looking at and paying attention to what the others are saying. It's challenging - each worker has to decide what to to say about their piece and what to pay attention to from others - but they work out a system. "I think this has something to do with whiskers, and I'm looking for something about whether the cat's in a small or large space". "I have information about the cat being alone and I'm trying to see if they're scared or not". "My tail is low between my legs, does anyone have information about a threat?"

A lot of what they communicate isn't useful, so they don't get rewarded for it. The workers focus on the information that gives them the most rewards, and because the rewards come from the next-word prediciton being right the whole system starts producing internal information that leads to better predictions.

Your training has gotten to the point where you're barely losing any rewards. You want to make sure people aren't just memorizing the whole cat communication book, so you kept one part of the book secret during training, and you use that part to test how good your next-word prediction is. It's looking good.

You have successfully built a Large (Cat) Language Model.

Transformers and Attention

One of the key breakthroughs that enabled the creation of sophisticated language models was the invention of transformers, which is the technical term for what we called attention above. If you're interested in the details and examples I'd recommend the illustrated transfomer and visualizing attention, both do a great job of giving you the math and intuition.

I'll steal one of the illustrated transfomer's images to give you a real world example of what attention looks like - the image shows what one of the "workers" is paying attention to in the sentence "The animal didn't cross the street because it was too tired":

The worker here is working on the word it, and the diagram shows it is paying the most attention to The and animal. That makes sense: in the above sentence it is referring to The animal, and the network has found a way allow that pronoun to be aware of its noun.

Cat Communication

You decide to try it out with the alien cats.

The cat representative is sitting across from you. It says something. You hand that over to your team of colleagues and they start their process of creating a response, one word at a time.

You give your response to the cat.

It gives you a surprised look, and raises its tail in a let's-be-friends gesture.

Back and forth, you try several exchanges, and it seems to be working - you're able to have a "conversation". You're not sure what you or the cat are saying, but you're conversing.

You notice some of your responses seem to upset the cat. You'd like to avoid that, so you start giving feedback to your team, increasing rewards for responses that make the cat happy and reducing rewards for responses that upset the cat. After a while your team learns to provide its responses in a way that makes the cat happy. You call this Cat-in-Loop-Training.

Human-in-the-Loop and Fine Tuning

ChatGPT is the fastest growing application in history - it went from zero to 1 million users in 5 days and to 100 million users in just two months.

But did you know that GPT-3, the base language model that was subsequently improved to power ChatGPT, was released in June 2020? What happend in those 2.5 years - why didn't GPT-3 capture our imagination the way ChatGPT did?

One of the main reasons was the "fine tuning" of the model based on human feedback to make it conversational and helpful.

The base language model, GPT-3, was trained on a very large amount of text from very many sources. This taught the system to be able to create cohesive sentences and paragraphs. Think of this as a computer that has learned to speak.

The human-in-the-loop fine tuning took this base system and further trained it to have chat-like conversations: the sentences created by the system were no longer just judged on how cohesive and valid they were, but also on how well they mimicked a helpful conversation.

ChatGPT is polite and helpful because it was trained to be that way - the language model itself is capable of many forms of communication, including being rude and unhelpful, since the very large amount of text it was trained on included rude and unhelpful conversations. ChatGPT was specifically trained to be a helpful chatbot, as opposed to a sarcastic layabout.

This idea of taking a base language model and further training it ("fine tuning" it) for your specific needs is very powerful: instead of having to create a complete model from scratch, you can take existing models off the shelf and tune them to do what you want.

(Another significant reason ChatGPT was so successful while GPT-3 was ignored was its easy to use and easy to understand user interface, but we'll conveniently ignore that here)

CatGPT

Congratulations, you have saved humanity, communicated with aliens, and perhaps understood a bit more about large language models and ChatGPT.

A Non-Technical Explanation of ChatGPT

2023-05-04T08:10:00-07:00

Continuing the series of non-technical explainers, let's figure out how ChatGPT (and in general Large Language Models, or LLMs) work. This is part one of the ChatGPT explainer, with part two coming soon.

It's helpful but not necessary to read the non-technical explanation of AI first, if you feel like it read that and come back.

Fill In The Blank

Let's play a quick game of fill in the blank:

to be or not to _____

Why did you immediately think of the word be to fill in that blank, instead of the word banana or fish? Because you've seen the phrase many times and your brain has learned the most likely next word is be.

Let's do another one:

rock and ____

Did you think of the word roll? Why? Because that's the word you see most often following the words rock and.

How about this one:

I am very _____

This is less clear - the next word depends on the context.

I just ran a marathon. I am very _____

Perhaps now you would say tired. Or happy, or proud.

Your brain is picking the most likely next word based on the context of the sentence and based on what words it's seen most frequently in that context.

Aliens and CatGPT

In an astonishing turn of events a race of alien cats have landed on earth. Since you are the world's foremost cat expert you have been chosen to communicate with them.

Your boss walks in and drops a massive print out of all the alien cat communications on your desk.

"Speak cat!" she commands.

Unfortunately you don't speak alien cat.

You page through the print outs - it looks like pages and pages of gibberish.

zoog zeeg zag. zoog zeeg kaz. zoog zeeg bah. rag zoog zeeg. rag zoog ko. kaz rag. zap zoog zeeg. ...

Markov to the Rescue

What to do? You remember your friend Markov who's always talking about languages, words, and their relationships. Maybe he can help. You show him the alien cat communications and ask him if he can help you say something in cat.

"Yes!" he exclaims - "We can do this. We will create our response one word at a time, just by picking the best word to say next, and keep going until we have a sentence"

"But how do we know what the best word to say next is? Don't we need to know what the words mean?"

"Nope, we just need to know what word to say next"

"That... doesn't seem like it would work"

"Let me show you", says Markov, and grabs the book on his desk, Alice in Wonderland.

"We just need to know what word to say next. The best word to say next is simply the word that shows up most frequently after the word we're looking at. All we have to do is make a table of how often each word follows another word. We'll call this a frequency table"

He shows you how to create the frequency table - you just note down how many times each word follows another. It takes a while to go through all Alice in Wonderland and do the counting. You end up with:

Next word after "Alice" → was: 17 times,  and: 16 times,  thought: 12 times,  had: 11 times, ...
Next word after "sat" → down: 9 times,  silent: 2 times,  still: 2 times,  for: 1 times, ...
Next word after "was" → a: 31 times,  the: 20 times,  not: 12 times,  going: 11 times, ...
Next word after "and" → the: 80 times,  she: 52 times,  then: 31 times,  was: 19 times, ...
Next word after "a" → little: 59 times,  very: 25 times,  large: 20 times,  great: 17 times
Next word after "very" → much: 10 times,  soon: 7 times,  curious: 6 times,  glad: 5 times
...

This table gives you a lot of help in forming sentences in the style of Alice in Wonderland. For example, if you start with the word Alice, then you'd look that up in the above table and see that the most frequent next word is was. And after was you would select a. After a you would get little. You'd end up with Alice was a little.

How well does this work? Let's create some sentences:

the door, when i beg your verdict," it was quite plainly through the bottle, i'm afraid that they had its nest.

You can try this out for yourself on almeopedia.

That's not great. Why is it so nonsensical? Think back to our fill in the blank examples at the start of this post: in order to pick a good next word you need context. If someone asked you what word should appear after rock, you'd have a hard time picking something reasonable, but if they gave you rock and, you'd fairly quickly think of roll.

What would happen if you considered two words instead of a single word for your context? For one thing your frequency table creation would become harder - now instead of a single line for each word, you'd have a line for each word pair. You'd have to gather stats for all the two word permutations, which is a lot more than the single word case.

Next word after "Alice was" → not: 3 times,  beginning: 2 times,  very: 2 times, ...
Next word after "was not" → a: 3 times,  here: 1 times,  going: 1 times ...
Next word after "not here" → before: 1 times
Next word after "not a" → moment: 2 times,  bit: 2 times,  serpent: 2 times

...

Let's create a sentence with this and see how it looks:

the hatter, and, burning with curiosity, she decided on going into the garden.

You can try this out for yourself on almeopedia.

That's looking better, it almost makes sense. Let's keep going - instead of two words of context, how about three?

The Hatter’s remark seemed to have no sort of chance of her ever getting out of the water, and seemed to quiver all over with diamonds, and walked two and two, as the soldiers did.

Four?

So she swallowed one of the cakes, and was delighted to find that she knew the name of nearly everything there.

Hmm. This is a little too good. It turns out a very similar sentence exists in the original Alice in Wonderland text:

So she swallowed one of the cakes, and was delighted to find that she began shrinking directly.

Our simple method of picking the most likely next word can result in the system memorizing the text snippets - given long enough context, the next most likely word is exactly the word that appeared in the original text following that context.

We can fix this by introducing some randomness - instead of always picking the most likely next word, we can pick somewhat likely next words. This way we're less likely to regurgitate the original text.

The good news is our method seems to work for English, so it'll probably work for alien cat language as well.

Aside: You might run into the term "stochastic" as you look into language models - this just means randomly determined. For example, you might hear people argue whether these systems are Stochastic Parrots, implying the systems are simply parroting back the original text that they saw, with some randomness thrown in.

Large Language Models

ChatGPT and many other Large Language Models (LLMs) do essentially what you did above: they examine a very large amount of human communication, gather stats (or probabilities) on what words are most likely to follow other words, and play a continous game of fill-in-the-blank. You give them some context (your prompt or question), and they create the reply one word at a time by selecting the word most likely to appear next. They respond with a word, then look at their internal stats to pick a word to follow their first word, then a word to follow their second word, and so forth, one word at a time, until they've formed a response.

To be Continued...

In part two of this series we'll look at the problems you'll run into with this method and how deep learning helps you overcome those problems.

The Parable of the Physics Professor

2023-04-21T23:38:00-07:00

One of my undergrad physics professors gave notoriously difficult exams, including an extra-point question that was truly challenging.

A brainiac friend of mine managed to get the extra-point question completely correct. Almost. He screwed up the decimal point, incorrectly putting it two places to the right.

The professor gave him zero points.

My friend was incensed - he did all of the many steps of this very difficult problem correctly, just misplaced the decimal point, and got zero points.

We went to the professor to appeal. The professor's response was: the answer to this question is a temperature. The temperature you got is hotter than the surface of the sun. As a physicist you should've immediately known that's not the right answer. I'm giving you zero points because the point is not to do the steps, the point is to understand the world.

A Completely Non-Technical Explanation of AI and Deep Learning

2023-04-14T23:38:00-07:00

Overview

This document will explain what neural networks are and how they work, which will help you understand how AI and machine learning work. In the scenario below you'll play the part of the neural network.

Day One

First day of your new job as a "classifier" your boss walks in and drops a big spreadsheet of numbers on your desk.

"Is this a cat?" she asks.

Confused you ask "What?"

"Is this a cat?"

Even more confused, you respond "I don't know".

"Wrong!" she says, and slaps you across the face.

Before you've had a chance to be shocked she drops another large spreadsheet on your desk. "Is this a cat?"

"I don't understand" you respond.

"Wrong!" she says, and slaps you again.

Another spreadsheet. "Is this a cat?"

Not wanting another slap, you meekly respond "Yes?"

"Correct! Good job!" she says, and gives you a wonderful reward. Almost makes up for the slaps.

Another spreadsheet. Cat?

Slightly more confident and wanting another reward you respond "Yes"

"Wrong!". Another slap.

You are very confused. "I don't understand what's going on. You haven't told me the rules, you haven't told me how to figure out something is a cat, you haven't given me the logic to figure this out. You haven't trained me."

"Correct" says the boss. "This is training. Trust me, you're going to get very good at recognizing cats"

Another spreadsheet. This time you focus on the sheet. It's 256 columns and 256 rows, filled with numbers. Is it a cat? You don't know, so you guess.

Another and another. Right, wrong, wrong, right again, it keeps going. Slaps and rewards.

You're starting to notice some patterns - if the sheet is almost all zeros then it's not a cat. You look at the center of the sheet - that part should have larger numbers.

It's getting slightly better - you're getting more rewards than slaps, guessing correctly more often than not. It's all about the patterns of the numbers.

Day Two

This is the strangest job you've ever had. The slaps are terrible, but the rewards are so great you don't want to quit. How do you get better at this? It's not really possible for one person. If you had more people they could focus on different aspects of the sheet, look for different patterns, and you could use their findings to make better guesses.

You hire 10 people and bring them to work the next day. When you get a spreadsheet you show it to those 10 people and ask them "Is this a cat?"

They are as confused as you were. You force them to guess. The first guy looks like an idiot, so you decide to go with the opposite of what he says. The third lady looks really thoughtful so you put a lot of weight on what she says. In your mind you assign a weight to each of their guesses to come up with your final answer.

Each time you get a reward or punishment you share it with the 10 people you've hired: you reward or slap them based on how much weight you put on their input and how much they contributed to you getting the answer right or wrong. You learn the patterns: if the first guy says a strong no and the third lady a strong yes then it's very likely a cat. You learn many more patterns like this.

The 10 people are learning to hone their opinions based on the rewards and punishments they get. They can focus on different aspects of the sheet, and their opinions together can tell you a lot about the sheet.

After what feels like an epoch and a lot of spreadsheets, eventually you get better. You're getting more rewards. This is working. Your boss says you're very perceptive, and starts calling you Perceptron.

Day Ten

What if you get more people involved? You could make it 50 people reporting to you, but that'd be hard to manage. How about we add another layer of people before your 10?

You hire 50 more people, have them give their guesses to the 10 people that report to you. You're now very removed from looking at the spreadsheets - instead you rely on the patterns found by the first layer of the people, who give their opinions to the second layer of the people, who then inform you. Everybody passes the slaps and rewards down the line based on how much each earlier person's guess contributed to their guess.

It takes even longer, but eventually the system starts working. You're much more accurate in identifying when it's a cat.

Interestingly you were never taught the rules or logic, and you didn't teach your people the logic or rules. You just propagated the rewards and punishments back through each layer: the more each person's opinion contributes to the answer, the more rewards or punishment you shared with them, and they in turn with the people in the layer behind them. Each person uses the same method to propogate their rewards and punishment back to the people in the layer below them.

Neural Networks

This is how neural networks work: they see many examples and get rewarded or punished based on whether their guesses are correct. They use multiple layers of workers and eventually learn patterns. Importantly no one is teaching them what patterns they should be looking for or telling them the logic or the rules - the networks eventually figure out the patterns and logic based on very many rounds of example, reward, and punishment. This is called machine learning, because the machine is learning the rules by itself.

Neural networks work well when you have many examples of something (eg. pictures of cats), but it's hard to write the logic and rules to describe how to recognize that thing. Try it - write down some rules for how to recognize a cat (eg. "has 4 legs"), then look at pictures of cats and see where the rules fail (eg. a picture of a cat's head). In these cases you use machine learning so the machine learns the rules by itself.

Also note that computers see things as multi-dimensional tables of data. They don't look at a "picture" - they see 3 spreadsheets of numbers representing the RGB values of the picture.

Day Forty

Identifying cats is a lucrative business and you're pretty good at it. But not good enough. How can we get even better? More people, more layers!

Unfortunately it takes a long time for people to do their calculations, so you've been stuck with just a few layers of people for a while. Having a lot more people would make the process take too long.

One day you run into a group of people who call themselves Gaming People United (GPU). These people play a game that has taught them to be very good at looking at spreadsheets and calculating numbers - in fact they can do a lot of calculations in parallel, very quickly.

Excited, you hire a bunch of them and put them to work recognizing cats. Now instead of two layers of people you can have 10! Each layer can focus on higher and higher concepts - the first layer can look for small details (eg. do I see a pattern that looks like a small circle? Do I see a pattern that looks like a sharp edge?), the second layer can look for patterns from the results of the first layer (eg. are there two circles close to each other), the third layer can build on that (eg. are there two circle close to each other, and a triangle below them), and so forth. By the time the results of the layers get to you you have some fairly sophisticated concepts - we see a pattern that looks like a face, we also see patterns that could be legs, and a sharp thing that might be a beak. Now your guess is more informed than ever.

You train the layers sending slaps and rewards down through each layer, and after many epochs you get really good at recognizing cats.

Deep Learning

One of the major break-throughs in machine learning was the advent of Deep Learning, which is basically what we describe above - GPUs (Graphics Processing Units) got popular because they enable fast 3D graphics for games. They also happen to be very good at quickly doing the types of calculations neural networks need. Machine learning people started using GPUs for training neural networks, and with this extra speed they could have many more layers - from 3 or 4 layers to 10 or 11, and then hundreds.

This addition of layers led to significant advances in neural network performance - very quickly neural networks became the best solution for voice recognition, for image recognition, image creation, and sophisticated language models. This is called Deep Learning because there are so many layers, not because it's profound.

You can stop reading here if you want, the following portion is only here because someone asked me what "convolutional neural networks" are.

Day Fifty

How can we make the process even more efficient? You start to realize that looking at the entire sheet is too hard - the first layer of people literally have to stare at the whole thing and try to guess based on that. What if we give them a small portion of the sheet to look at, give a guess for that portion, then move on to the next portion of the sheet, and so on? The first layer of people are focused on lower level concepts anyway - looking for edges, things that look like circles, and so forth - and they can find those in the small portions of the sheet, without the need to look at the whole thing at once.

Via towardsdatascience

You tell the first layer of people this is what they should start doing. They respond that this sounds very convoluted and they'll only agree to do it if you call them "colonel".

This makes the process even faster and more accurate: the first layer become specialists in small features of the sheet, and they learn to be very efficient and fast. You try the same idea for the next several layers: you tell people to focus on small portions of the feedback they get from the layer below them, moving the portion they focus on so eventually the scan across all of the feedback.

This specialization and focus makes you even more accurate. Congratulations, you are officially recognized as the best cat classifier in the world.

Convolutional Neural Networks

Convolutions are this idea of applying a specific set of calculations, sometimes called a kernel, to each portion of the input, and scanning your window of attention across the entire image. The set of calculations, or kernel, is learned by each worker - you don't tell it what to calculate or how, it learns what's useful based on rewards and punishments. Convolutional Neural Networks (CNNs) were a significant step forward in the capability of neural networks.

A Practical Introduction to Polars for Python Data Processing

2023-02-18T23:33:00-08:00

Polars is a "lightning-fast DataFrame library for Rust and Python" - if you're familiar with Pandas, Polars is quite similar but significantly faster and with a cleaner API. This post is a brief tour of how to set up and use Polars, working through a pragmatic example of the things you might do in a typical data exploration project.

The Setup

Install polars and jupyter:

python3 -m venv venv
source venv/bin/activate
pip install polars notebook

This sets up a python virtual environment, then installs the Polars library as well as jupyter. The page you're looking at was created as a jupyter notebook.

The Data

I'm using the New York Taxi Dataset parquet files in this notebook. I downloaded the October 2022 Yellow Taxi Trip Records (PARQUET) file, feel free to grab that as well.

The data is in parquet format, which is quite a nice format for data processing, significantly better than CSV.

Polars

Let's import polars and read the file:

import polars as pl

fname = "yellow_tripdata_2022-10.parquet"
data = pl.read_parquet(fname)

Let's see what it looks like

data.head(5)

shape: (5, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
1	2022-10-01 00:03:41	2022-10-01 00:18:39	1.0	1.7	1.0	"N"	249	107	1	9.5	3.0	0.5	2.65	0.0	0.3	15.95	2.5	0.0
2	2022-10-01 00:14:30	2022-10-01 00:19:48	2.0	0.72	1.0	"N"	151	238	2	5.5	0.5	0.5	0.0	0.0	0.3	9.3	2.5	0.0
2	2022-10-01 00:27:13	2022-10-01 00:37:41	1.0	1.74	1.0	"N"	238	166	1	9.0	0.5	0.5	2.06	0.0	0.3	12.36	0.0	0.0
1	2022-10-01 00:32:53	2022-10-01 00:38:55	0.0	1.3	1.0	"N"	142	239	1	6.5	3.0	0.5	2.05	0.0	0.3	12.35	2.5	0.0
1	2022-10-01 00:44:55	2022-10-01 00:50:21	0.0	1.0	1.0	"N"	238	166	1	6.0	0.5	0.5	1.8	0.0	0.3	9.1	0.0	0.0

What are the columns in this file?

data.columns

['VendorID',
 'tpep_pickup_datetime',
 'tpep_dropoff_datetime',
 'passenger_count',
 'trip_distance',
 'RatecodeID',
 'store_and_fwd_flag',
 'PULocationID',
 'DOLocationID',
 'payment_type',
 'fare_amount',
 'extra',
 'mta_tax',
 'tip_amount',
 'tolls_amount',
 'improvement_surcharge',
 'total_amount',
 'congestion_surcharge',
 'airport_fee']

Pretty easy to guess what the columns mean.

Let's poke around a little bit - what's the average taxi fare in NYC?

data["total_amount"].mean()

22.247540895418666

I wonder how many passengers there are in each ride?

data["passenger_count"].mean()

1.3846936672698376

How about the average fare from the airport? I noticed the airport_fee column, let's look at average total_amount when the airport fee is greater than zero:

data.filter( pl.col('airport_fee') > 0 )["total_amount"].mean()

55.83249747837022

How about the average non-airport ride cost?

data.filter( pl.col('airport_fee') == 0 )["total_amount"].mean()

18.957161390687517

Most expensive rides

Let's see what the most expensive fares in this data set are. We'll sort the data by total_amount and look at the top 10.

most_expensive = data\
    .sort("total_amount", reverse=True)\
    .head(10)\
    .select(["total_amount", "PULocationID", "DOLocationID"])

most_expensive

shape: (10, 3)

total_amount	PULocationID	DOLocationID
f64	i64	i64
1301.85	39	265
950.3	226	226
950.3	260	260
870.6	265	265
800.3	140	229
750.3	265	265
750.3	265	265
639.8	68	193
636.55	138	138
606.36	4	64

I'm selecting just the 3 columns I'm interested in so it's easier to look at the data - the amount and pick up and drop off locations.

Note the coding style above: I split the query into separate lines so it's easier to read. I've become a stickler when it comes to query / code style - you'll thank yourself for keeping it clean when you come back months later and try to figure out what you were doing.

We can make it even cleaner if we use the paranthesis syntax:

most_expensive = (
    data
    .sort("total_amount", reverse=True)
    .head(10)
    .select(["total_amount", "PULocationID", "DOLocationID"])
)

I wasn't crazy about the above syntax at first, but once I started using it I really liked it.

Also notice the style of computation here - we're chaining the actions, one after another, javascript style: sort, then take 10 rows, then select the columns we want. This I also like.

Looking at the data, apparently somebody paid \$1,301.85 for a single ride. I wonder where they went to? Let's find out.

PULocationID and DOLocationID are the pick up and drop off location IDs. They're just numbers right now, not too informative. Let's convert them to names.

Zone Names

We need a mapping from the Zone IDs to the Zone names. From the NYC TLC Data page, look for the Taxi Zone Lookup Table (CSV) link and download the CSV file.

zone_data = pl.read_csv("taxi+_zone_lookup.csv")
zone_data.head(10)

shape: (10, 4)

LocationID	Borough	Zone	service_zone
i64	str	str	str
1	"EWR"	"Newark Airport...	"EWR"
2	"Queens"	"Jamaica Bay"	"Boro Zone"
3	"Bronx"	"Allerton/Pelha...	"Boro Zone"
4	"Manhattan"	"Alphabet City"	"Yellow Zone"
5	"Staten Island"	"Arden Heights"	"Boro Zone"
6	"Staten Island"	"Arrochar/Fort ...	"Boro Zone"
7	"Queens"	"Astoria"	"Boro Zone"
8	"Queens"	"Astoria Park"	"Boro Zone"
9	"Queens"	"Auburndale"	"Boro Zone"
10	"Queens"	"Baisley Park"	"Boro Zone"

Ok, that's helpful, but I'd rather have the Zone and Borough together in a single column so it has all the info I need.

zone_data = (
    zone_data
    .with_column(
        (pl.col("Borough") + ": " + pl.col("Zone")).alias("Location"),
    )
    .select(["LocationID", "Location"])
)

zone_data.head(10)

shape: (10, 2)

LocationID	Location
i64	str
1	"EWR: Newark Ai...
2	"Queens: Jamaic...
3	"Bronx: Allerto...
4	"Manhattan: Alp...
5	"Staten Island:...
6	"Staten Island:...
7	"Queens: Astori...
8	"Queens: Astori...
9	"Queens: Auburn...
10	"Queens: Baisle...

Ok, that's looking pretty good. Now let's look at those expensive rides again, but add in the pick up and drop off location names.

To do this I'll join the most_expensive dataframe I created above, which just had the fare and pick up and drop off IDs, with the zone_data dataframe which has the zone names. First I'll join the PULocationID with the zone data, add the location name, and rename the column from Location to PickUpLocation, then I'll do the same with the DOLocationID.

The left join below means use the rows from the left dataframe, which is the most_expensive dataframe, and injects into those rows the Location names from the zone_data dataframe, joining on LocationID.

most_expensive_rides = (
    most_expensive
    .join(zone_data, how="left", left_on="PULocationID", right_on="LocationID").rename({"Location": "PickUpLocation"})
    .join(zone_data, how="left", left_on="DOLocationID", right_on="LocationID").rename({"Location": "DropOffLocation"})
)
most_expensive_rides

shape: (10, 5)

total_amount	PULocationID	DOLocationID	PickUpLocation	DropOffLocation
f64	i64	i64	str	str
1301.85	39	265	"Brooklyn: Cana...	"Unknown: NA"
950.3	226	226	"Queens: Sunnys...	"Queens: Sunnys...
950.3	260	260	"Queens: Woodsi...	"Queens: Woodsi...
870.6	265	265	"Unknown: NA"	"Unknown: NA"
800.3	140	229	"Manhattan: Len...	"Manhattan: Sut...
750.3	265	265	"Unknown: NA"	"Unknown: NA"
750.3	265	265	"Unknown: NA"	"Unknown: NA"
639.8	68	193	"Manhattan: Eas...	"Queens: Queens...
636.55	138	138	"Queens: LaGuar...	"Queens: LaGuar...
606.36	4	64	"Manhattan: Alp...	"Queens: Dougla...

The location names are getting cut off when we view the dataframe - let's configure the output to show wider columns

pl.Config. set_fmt_str_lengths(100)

most_expensive_rides

shape: (10, 5)

total_amount	PULocationID	DOLocationID	PickUpLocation	DropOffLocation
f64	i64	i64	str	str
1301.85	39	265	"Brooklyn: Canarsie"	"Unknown: NA"
950.3	226	226	"Queens: Sunnyside"	"Queens: Sunnyside"
950.3	260	260	"Queens: Woodside"	"Queens: Woodside"
870.6	265	265	"Unknown: NA"	"Unknown: NA"
800.3	140	229	"Manhattan: Lenox Hill East"	"Manhattan: Sutton Place/Turtle Bay North"
750.3	265	265	"Unknown: NA"	"Unknown: NA"
750.3	265	265	"Unknown: NA"	"Unknown: NA"
639.8	68	193	"Manhattan: East Chelsea"	"Queens: Queensbridge/Ravenswood"
636.55	138	138	"Queens: LaGuardia Airport"	"Queens: LaGuardia Airport"
606.36	4	64	"Manhattan: Alphabet City"	"Queens: Douglaston"

The Unknown: NA locations are not very useful, let's get rid of them. I'll filter the data so they're excluded.

Location ID of 256 means Unknown: NA, so we need to get rid of any row that has 256 in the PULocationID or DOLocationID columns.

clean_data = (
    data
    .filter( 
        (pl.col("PULocationID") != 265) &
        (pl.col("DOLocationID") != 265)
    )
)
clean_data.head(5)

shape: (5, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
1	2022-10-01 00:03:41	2022-10-01 00:18:39	1.0	1.7	1.0	"N"	249	107	1	9.5	3.0	0.5	2.65	0.0	0.3	15.95	2.5	0.0
2	2022-10-01 00:14:30	2022-10-01 00:19:48	2.0	0.72	1.0	"N"	151	238	2	5.5	0.5	0.5	0.0	0.0	0.3	9.3	2.5	0.0
2	2022-10-01 00:27:13	2022-10-01 00:37:41	1.0	1.74	1.0	"N"	238	166	1	9.0	0.5	0.5	2.06	0.0	0.3	12.36	0.0	0.0
1	2022-10-01 00:32:53	2022-10-01 00:38:55	0.0	1.3	1.0	"N"	142	239	1	6.5	3.0	0.5	2.05	0.0	0.3	12.35	2.5	0.0
1	2022-10-01 00:44:55	2022-10-01 00:50:21	0.0	1.0	1.0	"N"	238	166	1	6.0	0.5	0.5	1.8	0.0	0.3	9.1	0.0	0.0

Ok, now we can look at the most expensive rides again, with the Unknown locations removed

most_expensive = (
    clean_data
    .sort("total_amount", reverse=True)
    .head(10)
    .select(["total_amount", "PULocationID", "DOLocationID"])
)

most_expensive

shape: (10, 3)

total_amount	PULocationID	DOLocationID
f64	i64	i64
950.3	226	226
950.3	260	260
800.3	140	229
639.8	68	193
636.55	138	138
606.36	4	64
590.05	238	1
550.3	100	48
505.3	76	76
502.8	79	132

Let's add the location names back in. I'll create a function to do this so we can use it whenever we need.

def add_location_names(data):
    return (
        data
        .join(zone_data, how="left", left_on="PULocationID", right_on="LocationID").rename({"Location": "PickUpLocation"})
        .join(zone_data, how="left", left_on="DOLocationID", right_on="LocationID").rename({"Location": "DropOffLocation"})
    )

add_location_names(most_expensive)

shape: (10, 5)

total_amount	PULocationID	DOLocationID	PickUpLocation	DropOffLocation
f64	i64	i64	str	str
950.3	226	226	"Queens: Sunnyside"	"Queens: Sunnyside"
950.3	260	260	"Queens: Woodside"	"Queens: Woodside"
800.3	140	229	"Manhattan: Lenox Hill East"	"Manhattan: Sutton Place/Turtle Bay North"
639.8	68	193	"Manhattan: East Chelsea"	"Queens: Queensbridge/Ravenswood"
636.55	138	138	"Queens: LaGuardia Airport"	"Queens: LaGuardia Airport"
606.36	4	64	"Manhattan: Alphabet City"	"Queens: Douglaston"
590.05	238	1	"Manhattan: Upper West Side North"	"EWR: Newark Airport"
550.3	100	48	"Manhattan: Garment District"	"Manhattan: Clinton East"
505.3	76	76	"Brooklyn: East New York"	"Brooklyn: East New York"
502.8	79	132	"Manhattan: East Village"	"Queens: JFK Airport"

Hmm. Somebody paid \$950.3 to go from Queens: Sunnyside to Queens: Sunnyside. Same for Woodside. There was also an \$800 ride from Lenox Hill East to Sutton Place, which according to Google is a 4 minute car ride.

I'm starting to think our data might have junk in it. Exciting - finding junk in you data means you're doing Data Science™. Now we just have to throw in some Excel and we'll have Machine Learning™.

There's a lot we could do from here. For example, we could calculate the average fare by pick up and drop off location. Or we can look at the impact of pick up time on fares (there's a tpep_pickup_datetime column in the data).

For some reason I decided it'd be interesting to see which pickup-dropoff location pairs have the highest variation in fare. We can figure this out by grouping the data by ("PULocationID", "DOLocationID"), then calculating the standard deviation for each grouping, and looking at the locations with the highest deviation.

amount_stddevs = (
    clean_data
    .groupby(["PULocationID", "DOLocationID"])
    .agg(
        [
            pl.col("total_amount").std().alias("total_amount_stddev")
        ]
    )
    .sort("total_amount_stddev", reverse=True)
    .head(20)

)

amount_stddevs

shape: (20, 3)

PULocationID	DOLocationID	total_amount_stddev
i64	i64	f64
16	175	219.899498
192	129	190.126871
260	223	155.620968
93	192	137.885822
98	98	137.461869
194	129	137.14336
55	132	130.249069
50	182	129.223764
260	129	122.231773
206	206	120.172797
68	193	119.954706
250	250	119.445285
19	19	118.451447
85	28	115.611959
137	201	114.268456
223	174	111.510739
92	252	110.662211
132	184	107.047076
164	187	104.368961
209	117	103.060813

Let's add the location names:

add_location_names(amount_stddevs)

shape: (20, 5)

PULocationID	DOLocationID	total_amount_stddev	PickUpLocation	DropOffLocation
i64	i64	f64	str	str
16	175	219.899498	"Queens: Bayside"	"Queens: Oakland Gardens"
192	129	190.126871	"Queens: Queensboro Hill"	"Queens: Jackson Heights"
260	223	155.620968	"Queens: Woodside"	"Queens: Steinway"
93	192	137.885822	"Queens: Flushing Meadows-Corona Park"	"Queens: Queensboro Hill"
98	98	137.461869	"Queens: Fresh Meadows"	"Queens: Fresh Meadows"
194	129	137.14336	"Manhattan: Randalls Island"	"Queens: Jackson Heights"
55	132	130.249069	"Brooklyn: Coney Island"	"Queens: JFK Airport"
50	182	129.223764	"Manhattan: Clinton West"	"Bronx: Parkchester"
260	129	122.231773	"Queens: Woodside"	"Queens: Jackson Heights"
206	206	120.172797	"Staten Island: Saint George/New Brighton"	"Staten Island: Saint George/New Brighton"
68	193	119.954706	"Manhattan: East Chelsea"	"Queens: Queensbridge/Ravenswood"
250	250	119.445285	"Bronx: Westchester Village/Unionport"	"Bronx: Westchester Village/Unionport"
19	19	118.451447	"Queens: Bellerose"	"Queens: Bellerose"
85	28	115.611959	"Brooklyn: Erasmus"	"Queens: Briarwood/Jamaica Hills"
137	201	114.268456	"Manhattan: Kips Bay"	"Queens: Rockaway Park"
223	174	111.510739	"Queens: Steinway"	"Bronx: Norwood"
92	252	110.662211	"Queens: Flushing"	"Queens: Whitestone"
132	184	107.047076	"Queens: JFK Airport"	"Bronx: Pelham Bay Park"
164	187	104.368961	"Manhattan: Midtown South"	"Staten Island: Port Richmond"
209	117	103.060813	"Manhattan: Seaport"	"Queens: Hammels/Arverne"

That's... less interesting than I thought it would be. I suppose the absolute values are pretty interesting - our top hit, Queens: Bayside to Queens: Oakland Gardens, is about a 5 minute car ride and has a standard deviatiion of \$220. That seems suspect.

This much standard deviation means we have bad data.

Data Exploration and Cleansing

A lot of real-world data work involves cleaning data and finding outliers, so let's try that with this data.

There are a number of techniques for finding outliers, but first let's take a better look at our data. Polars has a built-in describe function that calculates basic statistics for your dataframe:

data.describe()

shape: (7, 20)

describe	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
str	f64	str	str	f64	f64	f64	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64
"count"	3.675411e6	"3675411"	"3675411"	3.675411e6	3.675411e6	3.675411e6	"3675411"	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6	3.675411e6
"null_count"	0.0	"0"	"0"	133021.0	0.0	133021.0	"133021"	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	133021.0	133021.0
"mean"	1.720443	null	null	1.384694	6.206977	1.424337	null	165.498365	162.891786	1.188275	15.299953	0.987793	0.487229	2.859254	0.583306	0.295531	22.247541	2.28062	0.103128
"std"	0.480032	null	null	0.930231	640.823768	5.685591	null	65.213026	70.197474	0.540651	14.674444	1.242828	0.101155	3.374175	2.111967	0.051038	18.368245	0.75784	0.347974
"min"	1.0	"2008-12-31 23:02:01.000000000"	"2009-01-01 02:20:59.000000000"	0.0	0.0	1.0	"N"	1.0	1.0	0.0	-1294.5	-22.18	-0.5	-100.01	-70.0	-0.3	-1301.85	-2.5	-1.25
"max"	6.0	"2022-11-01 01:27:35.000000000"	"2022-11-03 17:26:46.000000000"	9.0	389678.46	99.0	"Y"	265.0	265.0	4.0	1294.5	10.8	25.48	500.0	516.75	1.0	1301.85	2.5	1.25
"median"	2.0	null	null	1.0	1.9	1.0	null	162.0	162.0	1.0	10.5	0.5	0.5	2.22	0.0	0.3	16.55	2.5	0.0

That's pretty helpful - for example, the minimum pick up date is 2008-12-31, which means this data starts all the way back to 2009. Because of the way the TLC data download page is organized I thought I only had data from October 2023.

The largest tip amount is \$500 (max of the tip_amount column). The minimum tip is \$-100.01. Must've been a pretty rough ride.

Looking at the total_amount column we see that the maximum ride amount was \$1301.85 and the minimum was -\$1301.85. I don't know much about this data, but I'm going to guess that's an incorrectly charged ride and a refund. That's unfortunate - having large invalid data (\$1301.85 and -\$1301.85) at either extreme of values will wreak havoc on our standard deviations and such.

The longest ride (max of the trip_distance field) is 389678.46. Is that in miles? Let's look at the data dictionary provided by the TLC.

It is in fact in miles. The length of the United States is about 3000 miles, so this enterprising taxi travelled the entire US about 130 times.

The data dictionary tells us a lot of other useful things - for example, the Payment_type column should help us find disputed, voided, and other invalid charges.

Basic Cleaning

So what should we do? I'm thinking we look at more recent data, say anything 2020 and after. Let's get rid of anything with more than 3000 miles, and let's also get rid of rows with unreasonable negative numbers - eg. negative total amounts, tolls, etc. We can get rid of rows where the Payment_type is No Charge, Dispute, Voided, etc.

The trouble is if we just get rid of the negative amounts and disputes then we're leaving the corresponding equally invalid positive amount in the data - eg. if we drop the row that has the -\$1301.85 amount we'll still have the \$1301.85 row in our data.

We'll have to figure out how to drop both: for each of the negative numbers, we'll need to find the corresponding positive amount and get rid of those rows too.

Let's take a look at the outlier data and see if spot any clues for how to get rid of them.

(
    data
    .filter(
        (pl.col("total_amount") > 1300) |
        (pl.col("total_amount") < -1300)
    )
)

shape: (2, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
2	2022-10-12 07:00:43	2022-10-12 12:31:57	4.0	259.33	4.0	"N"	39	265	2	-1294.5	0.0	-0.5	0.0	-6.55	-0.3	-1301.85	0.0	0.0
2	2022-10-12 07:00:43	2022-10-12 12:31:57	4.0	259.33	4.0	"N"	39	265	2	1294.5	0.0	0.5	0.0	6.55	0.3	1301.85	0.0	0.0

The payment type is 2, which is cash. Somebody paid $1300 in cash, then got it handed back to him. I was hoping the payment type would be Voided or something reasonable.

Getting rid of these invalid pairs is going to be a bit painful - we'll have to find each invalid ride and also find its pair.

Before we do that, let's see if the problems still exists in data we care about, which is data from 2020 onwards.

data.filter( pl.col("tpep_pickup_datetime") > "2020" ).describe()

Warning: Comparing date/datetime/time column to string value, this will lead to string comparison and is unlikely what you want.
If this is intended, consider using an explicit cast to silence this warning.

shape: (7, 20)

describe	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
str	f64	str	str	f64	f64	f64	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64
"count"	3.675407e6	"3675407"	"3675407"	3.675407e6	3.675407e6	3.675407e6	"3675407"	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6
"null_count"	0.0	"0"	"0"	133021.0	0.0	133021.0	"133021"	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	133021.0	133021.0
"mean"	1.720442	null	null	1.384694	6.206973	1.424337	null	165.498365	162.891771	1.188275	15.299922	0.987794	0.487229	2.859248	0.583304	0.295531	22.247504	2.280621	0.103127
"std"	0.480032	null	null	0.930231	640.824117	5.685594	null	65.213049	70.197445	0.540651	14.674414	1.242828	0.101155	3.374169	2.111966	0.051038	18.368208	0.757839	0.347973
"min"	1.0	"2022-09-30 14:06:36.000000000"	"2022-09-30 14:23:04.000000000"	0.0	0.0	1.0	"N"	1.0	1.0	0.0	-1294.5	-22.18	-0.5	-100.01	-70.0	-0.3	-1301.85	-2.5	-1.25
"max"	6.0	"2022-11-01 01:27:35.000000000"	"2022-11-03 17:26:46.000000000"	9.0	389678.46	99.0	"Y"	265.0	265.0	4.0	1294.5	10.8	25.48	500.0	516.75	1.0	1301.85	2.5	1.25
"median"	2.0	null	null	1.0	1.9	1.0	null	162.0	162.0	1.0	10.5	0.5	0.5	2.22	0.0	0.3	16.55	2.5	0.0

That didn't work - our min tpep_pickup_datetime is 2022-09-30 (I was expecting 2020). I guess I can't filter a datetime column (tpep_pickup_datetime) using a string ("2020"). Makes sense, let's fix it.

from datetime import datetime
start_date = datetime(2020,1,1,0,0,0)
print(f"Filtering data to start at {start_date}")
data.filter( pl.col("tpep_pickup_datetime") > start_date ).describe()

Filtering data to start at 2020-01-01 00:00:00

shape: (7, 20)

describe	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
str	f64	str	str	f64	f64	f64	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64
"count"	3.675407e6	"3675407"	"3675407"	3.675407e6	3.675407e6	3.675407e6	"3675407"	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6	3.675407e6
"null_count"	0.0	"0"	"0"	133021.0	0.0	133021.0	"133021"	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	133021.0	133021.0
"mean"	1.720442	null	null	1.384694	6.206973	1.424337	null	165.498365	162.891771	1.188275	15.299922	0.987794	0.487229	2.859248	0.583304	0.295531	22.247504	2.280621	0.103127
"std"	0.480032	null	null	0.930231	640.824117	5.685594	null	65.213049	70.197445	0.540651	14.674414	1.242828	0.101155	3.374169	2.111966	0.051038	18.368208	0.757839	0.347973
"min"	1.0	"2022-09-30 14:06:36.000000000"	"2022-09-30 14:23:04.000000000"	0.0	0.0	1.0	"N"	1.0	1.0	0.0	-1294.5	-22.18	-0.5	-100.01	-70.0	-0.3	-1301.85	-2.5	-1.25
"max"	6.0	"2022-11-01 01:27:35.000000000"	"2022-11-03 17:26:46.000000000"	9.0	389678.46	99.0	"Y"	265.0	265.0	4.0	1294.5	10.8	25.48	500.0	516.75	1.0	1301.85	2.5	1.25
"median"	2.0	null	null	1.0	1.9	1.0	null	162.0	162.0	1.0	10.5	0.5	0.5	2.22	0.0	0.3	16.55	2.5	0.0

My min tpep_pickup_datetime is still 2022-09-30 (I was expecting 2020). What's going on? Maybe our data is only for Octover 2022 after all, and the 2008 date I was seeing was spurious? Note that we had 3675411 data points before we filtered by date and now we have 3675407 data points (I'm getting this by looking at the count field for tpep_pickup_datetime). Our date filtering only got rid of 4 data points. So we're looking only at October 2022 data with a couple of spurious data points thrown in.

I'm curious what the data with bad dates look like.

(
    data
    .filter(
        (pl.col("tpep_pickup_datetime") < start_date)
    )
)

shape: (4, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
2	2009-01-01 01:57:50	2009-01-01 02:20:59	1.0	7.84	1.0	"N"	161	264	2	22.0	0.5	0.5	0.0	0.0	0.3	23.3	0.0	0.0
2	2008-12-31 23:02:01	2009-01-01 19:08:45	1.0	22.93	1.0	"N"	132	25	1	63.0	0.0	0.5	13.01	0.0	0.3	78.06	0.0	1.25
2	2008-12-31 23:02:02	2009-01-01 20:14:39	1.0	0.0	2.0	"N"	231	231	1	52.0	0.0	0.5	10.0	0.0	0.3	66.55	2.5	1.25
2	2009-01-01 00:02:31	2009-01-01 17:02:49	1.0	8.11	1.0	"N"	138	186	1	35.5	1.0	0.5	9.52	6.55	0.3	57.12	2.5	1.25

Finding bad pairs

I don't see any obvious ways to correlate the bad pairs - the rows don't have IDs, there is no cab ID, and the payment method field doesn't give us much either. We'll have to data science.

Let's try this: - Find negative data points - For each negative data point, find a data point with the same amount but positive that also matches other parameters such as date and pick up / drop off location

(
    data
    .filter( pl.col("fare_amount") < 0)
)

shape: (25899, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
2	2022-10-01 00:52:07	2022-10-01 01:02:44	1.0	2.83	1.0	"N"	141	79	4	-10.5	-0.5	-0.5	0.0	0.0	-0.3	-14.3	-2.5	0.0
2	2022-10-01 00:29:57	2022-10-01 00:34:37	1.0	0.31	1.0	"N"	264	264	4	-4.5	-0.5	-0.5	0.0	0.0	-0.3	-8.3	-2.5	0.0
2	2022-10-01 00:31:37	2022-10-01 00:32:53	1.0	0.31	1.0	"N"	138	138	4	-3.0	-0.5	-0.5	0.0	0.0	-0.3	-5.55	0.0	-1.25
2	2022-10-01 00:46:38	2022-10-01 00:46:44	1.0	0.0	2.0	"N"	249	249	3	-52.0	0.0	-0.5	0.0	0.0	-0.3	-55.3	-2.5	0.0
2	2022-10-01 00:06:08	2022-10-01 00:48:23	1.0	17.05	1.0	"N"	238	175	4	-50.5	-0.5	-0.5	0.0	-6.55	-0.3	-60.85	-2.5	0.0
2	2022-10-01 00:40:58	2022-10-01 00:56:33	1.0	2.7	5.0	"N"	148	90	4	-27.31	0.0	-0.5	0.0	0.0	-0.3	-30.61	-2.5	0.0
2	2022-10-01 00:43:19	2022-10-01 00:44:02	1.0	0.02	1.0	"N"	90	90	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-6.3	-2.5	0.0
2	2022-10-01 00:12:32	2022-10-01 00:12:51	1.0	0.0	1.0	"N"	48	48	4	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-6.3	-2.5	0.0
2	2022-10-01 00:12:24	2022-10-01 00:13:06	1.0	0.33	1.0	"N"	138	138	4	-3.0	-0.5	-0.5	0.0	0.0	-0.3	-5.55	0.0	-1.25
2	2022-10-01 00:39:21	2022-10-01 00:40:07	4.0	0.15	1.0	"N"	230	230	3	-2.5	-0.5	-0.5	0.0	0.0	-0.3	-6.3	-2.5	0.0
2	2022-10-01 00:08:25	2022-10-01 00:21:22	2.0	1.46	1.0	"N"	79	211	4	-9.5	-0.5	-0.5	0.0	0.0	-0.3	-13.3	-2.5	0.0
2	2022-10-01 00:45:01	2022-10-01 01:18:01	2.0	2.35	1.0	"N"	79	249	4	-13.0	-0.5	-0.5	0.0	0.0	-0.3	-16.8	-2.5	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2	2022-10-06 16:17:49	2022-10-06 16:20:42	null	0.28	null	null	79	107	0	-95.68	0.0	0.5	15.22	0.0	0.3	-77.16	null	null
2	2022-10-10 12:27:10	2022-10-10 12:56:00	null	60043.58	null	null	125	68	0	-17.28	0.0	-0.5	-4.49	0.0	0.3	-24.77	null	null
2	2022-10-13 21:55:30	2022-10-13 21:56:26	null	0.03	null	null	144	144	0	-27.33	0.0	0.5	3.69	0.0	0.3	-20.34	null	null
2	2022-10-13 22:44:00	2022-10-13 22:45:00	null	0.03	null	null	90	90	0	-38.85	0.0	0.5	6.62	0.0	0.3	-28.93	null	null
2	2022-10-14 03:26:00	2022-10-14 03:27:00	null	0.04	null	null	114	114	0	-26.79	0.0	0.5	0.0	0.0	0.3	-23.49	null	null
2	2022-10-16 04:52:00	2022-10-16 05:01:00	null	2.71	null	null	233	262	0	-61.61	0.0	0.5	5.9	0.0	0.3	-52.41	null	null
1	2022-10-18 13:06:39	2022-10-18 13:06:52	null	0.0	null	null	232	232	0	-0.8	0.0	0.5	0.0	0.0	0.3	2.5	null	null
2	2022-10-19 01:55:00	2022-10-19 01:56:00	null	0.06	null	null	237	237	0	-22.39	0.0	0.5	2.92	0.0	0.3	-16.17	null	null
1	2022-10-22 19:35:59	2022-10-22 19:36:03	null	0.0	null	null	239	239	0	-0.8	0.0	0.5	0.0	0.0	0.3	2.5	null	null
2	2022-10-25 07:40:00	2022-10-25 07:42:00	null	0.07	null	null	140	140	0	-54.25	0.0	0.5	7.29	0.0	0.3	-43.66	null	null
1	2022-10-28 09:13:07	2022-10-28 09:13:26	null	0.0	null	null	141	141	0	-0.8	0.0	0.5	0.0	0.0	0.3	2.5	null	null
2	2022-10-29 07:33:24	2022-10-29 07:37:01	null	0.5	null	null	37	37	0	-18.55	0.0	0.5	2.72	0.0	0.3	-15.03	null	null

There's probably a clever way to do this, but my initial thought for getting rid of the corresponding positives is to create a new dataframe with the negative rows, then do an anti-join back to the original dataframe. An anti-join should find matching rows and remove them.

negative_data = (
    data
    .filter( pl.col("fare_amount") < 0)
).select(["tpep_pickup_datetime", "PULocationID", "fare_amount"])

negative_data

shape: (25899, 3)

tpep_pickup_datetime	PULocationID	fare_amount
datetime[ns]	i64	f64
2022-10-01 00:52:07	141	-10.5
2022-10-01 00:29:57	264	-4.5
2022-10-01 00:31:37	138	-3.0
2022-10-01 00:46:38	249	-52.0
2022-10-01 00:06:08	238	-50.5
2022-10-01 00:40:58	148	-27.31
2022-10-01 00:43:19	90	-2.5
2022-10-01 00:12:32	48	-2.5
2022-10-01 00:12:24	138	-3.0
2022-10-01 00:39:21	230	-2.5
2022-10-01 00:08:25	79	-9.5
2022-10-01 00:45:01	79	-13.0
...	...	...
2022-10-06 16:17:49	79	-95.68
2022-10-10 12:27:10	125	-17.28
2022-10-13 21:55:30	144	-27.33
2022-10-13 22:44:00	90	-38.85
2022-10-14 03:26:00	114	-26.79
2022-10-16 04:52:00	233	-61.61
2022-10-18 13:06:39	232	-0.8
2022-10-19 01:55:00	237	-22.39
2022-10-22 19:35:59	239	-0.8
2022-10-25 07:40:00	140	-54.25
2022-10-28 09:13:07	141	-0.8
2022-10-29 07:33:24	37	-18.55

I'm only grabbing a few columns for my negative_data dataframe, since all I'm going to use it for is to remove the corresponding positive rows from the original data. I'll try matching on the pick up date and pick up location.

Let's convert the negative amounts to positive so we can directly match them to their corresponding positive rows in the original data.

negative_data = negative_data.with_columns([
    (pl.col("fare_amount") * -1).alias("amount_positive")
])

negative_data

shape: (25899, 4)

tpep_pickup_datetime	PULocationID	fare_amount	amount_positive
datetime[ns]	i64	f64	f64
2022-10-01 00:52:07	141	-10.5	10.5
2022-10-01 00:29:57	264	-4.5	4.5
2022-10-01 00:31:37	138	-3.0	3.0
2022-10-01 00:46:38	249	-52.0	52.0
2022-10-01 00:06:08	238	-50.5	50.5
2022-10-01 00:40:58	148	-27.31	27.31
2022-10-01 00:43:19	90	-2.5	2.5
2022-10-01 00:12:32	48	-2.5	2.5
2022-10-01 00:12:24	138	-3.0	3.0
2022-10-01 00:39:21	230	-2.5	2.5
2022-10-01 00:08:25	79	-9.5	9.5
2022-10-01 00:45:01	79	-13.0	13.0
...	...	...	...
2022-10-06 16:17:49	79	-95.68	95.68
2022-10-10 12:27:10	125	-17.28	17.28
2022-10-13 21:55:30	144	-27.33	27.33
2022-10-13 22:44:00	90	-38.85	38.85
2022-10-14 03:26:00	114	-26.79	26.79
2022-10-16 04:52:00	233	-61.61	61.61
2022-10-18 13:06:39	232	-0.8	0.8
2022-10-19 01:55:00	237	-22.39	22.39
2022-10-22 19:35:59	239	-0.8	0.8
2022-10-25 07:40:00	140	-54.25	54.25
2022-10-28 09:13:07	141	-0.8	0.8
2022-10-29 07:33:24	37	-18.55	18.55

Now that we have a dataframe with the negative amounts converted to positive, we can join back to the original data, matching on the (now positive) fare amount and the pick up location. We'll do an anti-join, which gets rids of the rows that match. If Polars didn't have anti-join as an option we could have merged the new dataframe with the original data and dropped duplicate rows.

data.join(negative_data, on=("PULocationID", "fare_amount"), how="anti")

shape: (3649512, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
1	2022-10-01 00:03:41	2022-10-01 00:18:39	1.0	1.7	1.0	"N"	249	107	1	9.5	3.0	0.5	2.65	0.0	0.3	15.95	2.5	0.0
2	2022-10-01 00:14:30	2022-10-01 00:19:48	2.0	0.72	1.0	"N"	151	238	2	5.5	0.5	0.5	0.0	0.0	0.3	9.3	2.5	0.0
2	2022-10-01 00:27:13	2022-10-01 00:37:41	1.0	1.74	1.0	"N"	238	166	1	9.0	0.5	0.5	2.06	0.0	0.3	12.36	0.0	0.0
1	2022-10-01 00:32:53	2022-10-01 00:38:55	0.0	1.3	1.0	"N"	142	239	1	6.5	3.0	0.5	2.05	0.0	0.3	12.35	2.5	0.0
1	2022-10-01 00:44:55	2022-10-01 00:50:21	0.0	1.0	1.0	"N"	238	166	1	6.0	0.5	0.5	1.8	0.0	0.3	9.1	0.0	0.0
1	2022-10-01 00:22:52	2022-10-01 00:52:14	1.0	6.8	1.0	"Y"	186	41	2	25.5	3.0	0.5	0.0	0.0	0.3	29.3	2.5	0.0
2	2022-10-01 00:33:19	2022-10-01 00:44:51	3.0	1.88	1.0	"N"	162	145	2	10.5	0.5	0.5	0.0	0.0	0.3	14.3	2.5	0.0
1	2022-10-01 00:02:42	2022-10-01 00:50:01	1.0	12.2	1.0	"N"	100	22	1	41.0	3.0	0.5	3.0	0.0	0.3	47.8	2.5	0.0
2	2022-10-01 00:06:35	2022-10-01 00:24:38	1.0	7.79	1.0	"N"	138	112	1	23.5	0.5	0.5	4.96	0.0	0.3	31.01	0.0	1.25
2	2022-10-01 00:29:25	2022-10-01 00:43:15	1.0	4.72	1.0	"N"	145	75	1	14.5	0.5	0.5	1.5	0.0	0.3	19.8	2.5	0.0
1	2022-10-01 00:01:55	2022-10-01 00:20:16	1.0	8.8	1.0	"N"	138	236	1	26.0	4.25	0.5	5.64	6.55	0.3	43.24	2.5	1.25
1	2022-10-01 00:27:48	2022-10-01 00:59:50	1.0	8.6	1.0	"N"	140	36	1	29.5	3.0	0.5	6.0	0.0	0.3	39.3	2.5	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2	2022-10-31 23:30:42	2022-10-31 23:40:16	null	1.7	null	null	158	90	0	9.95	0.0	0.5	2.93	0.0	0.3	16.18	null	null
1	2022-10-31 23:42:05	2022-11-01 00:13:58	null	17.5	null	null	132	68	0	52.0	1.25	0.5	9.46	6.55	0.3	72.56	null	null
2	2022-10-31 23:52:34	2022-11-01 00:23:32	null	12.6	null	null	116	37	0	40.67	0.0	0.5	5.26	6.55	0.3	53.28	null	null
2	2022-10-31 23:16:12	2022-10-31 23:32:36	null	6.23	null	null	158	166	0	22.5	0.0	0.5	5.67	0.0	0.3	31.47	null	null
2	2022-10-31 23:15:00	2022-10-31 23:20:00	null	0.72	null	null	142	142	0	9.95	0.0	0.5	2.92	0.0	0.3	16.17	null	null
1	2022-10-31 23:20:21	2022-10-31 23:34:15	null	2.7	null	null	163	68	0	12.0	0.5	0.5	3.95	0.0	0.3	19.75	null	null
2	2022-10-31 23:45:37	2022-11-01 00:00:39	null	2.45	null	null	249	162	0	11.97	0.0	0.5	3.39	0.0	0.3	18.66	null	null
2	2022-10-31 23:56:35	2022-11-01 00:10:11	null	2.25	null	null	137	50	0	12.68	0.0	0.5	3.16	0.0	0.3	19.14	null	null
2	2022-10-31 23:22:00	2022-10-31 23:28:00	null	1.22	null	null	142	161	0	10.0	0.0	0.5	1.0	0.0	0.3	14.3	null	null
2	2022-10-31 23:25:00	2022-10-31 23:48:00	null	4.7	null	null	186	45	0	19.07	0.0	0.5	4.97	0.0	0.3	27.34	null	null
2	2022-10-31 23:32:54	2022-10-31 23:33:02	null	0.0	null	null	264	263	0	16.82	0.0	0.5	4.58	0.0	0.3	22.2	null	null
1	2022-10-31 23:21:34	2022-10-31 23:32:42	null	0.0	null	null	166	116	0	8.93	0.0	0.5	0.0	0.0	0.3	9.73	null	null

Did that work? Let's see how many rows that got rid of. After the anti-join we end up with 3649512 rows. We had and 25899 rows with negative values.

n_rows_original = data.shape[0]
n_rows_negative = negative_data.shape[0]
print(f"We are expecting {n_rows_original - n_rows_negative} rows if our removal worked perfectly")

We are expecting 3649512 rows if our removal worked perfectly

Sure enough the result of the anti-join has 3649512 rows (you can see that from the shape output a few cells above), so our method is working well. We should sanity check to make sure it's working as expected, but for now let's assume all is well.

We can now do our final cleaning: we'll get rid of the negative and corresponding positive rows, get rid of anything where the mileage is over 3000 miles, remove dates that are out of range, and get rid of rows with unknown pick up and drop off locations:

clean_data = (
    data
    .join(negative_data, on=("PULocationID", "fare_amount"), how="anti")
    .filter( pl.col("fare_amount") >= 0)
    .filter( pl.col("trip_distance") <= 3000 )
    .filter( pl.col("tpep_pickup_datetime") >= start_date )
    .filter( 
        (pl.col("PULocationID") != 265) &
        (pl.col("DOLocationID") != 265)
    )
)

clean_data

shape: (3630123, 19)

VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
i64	datetime[ns]	datetime[ns]	f64	f64	f64	str	i64	i64	i64	f64	f64	f64	f64	f64	f64	f64	f64	f64
1	2022-10-01 00:03:41	2022-10-01 00:18:39	1.0	1.7	1.0	"N"	249	107	1	9.5	3.0	0.5	2.65	0.0	0.3	15.95	2.5	0.0
2	2022-10-01 00:14:30	2022-10-01 00:19:48	2.0	0.72	1.0	"N"	151	238	2	5.5	0.5	0.5	0.0	0.0	0.3	9.3	2.5	0.0
2	2022-10-01 00:27:13	2022-10-01 00:37:41	1.0	1.74	1.0	"N"	238	166	1	9.0	0.5	0.5	2.06	0.0	0.3	12.36	0.0	0.0
1	2022-10-01 00:32:53	2022-10-01 00:38:55	0.0	1.3	1.0	"N"	142	239	1	6.5	3.0	0.5	2.05	0.0	0.3	12.35	2.5	0.0
1	2022-10-01 00:44:55	2022-10-01 00:50:21	0.0	1.0	1.0	"N"	238	166	1	6.0	0.5	0.5	1.8	0.0	0.3	9.1	0.0	0.0
1	2022-10-01 00:22:52	2022-10-01 00:52:14	1.0	6.8	1.0	"Y"	186	41	2	25.5	3.0	0.5	0.0	0.0	0.3	29.3	2.5	0.0
2	2022-10-01 00:33:19	2022-10-01 00:44:51	3.0	1.88	1.0	"N"	162	145	2	10.5	0.5	0.5	0.0	0.0	0.3	14.3	2.5	0.0
1	2022-10-01 00:02:42	2022-10-01 00:50:01	1.0	12.2	1.0	"N"	100	22	1	41.0	3.0	0.5	3.0	0.0	0.3	47.8	2.5	0.0
2	2022-10-01 00:06:35	2022-10-01 00:24:38	1.0	7.79	1.0	"N"	138	112	1	23.5	0.5	0.5	4.96	0.0	0.3	31.01	0.0	1.25
2	2022-10-01 00:29:25	2022-10-01 00:43:15	1.0	4.72	1.0	"N"	145	75	1	14.5	0.5	0.5	1.5	0.0	0.3	19.8	2.5	0.0
1	2022-10-01 00:01:55	2022-10-01 00:20:16	1.0	8.8	1.0	"N"	138	236	1	26.0	4.25	0.5	5.64	6.55	0.3	43.24	2.5	1.25
1	2022-10-01 00:27:48	2022-10-01 00:59:50	1.0	8.6	1.0	"N"	140	36	1	29.5	3.0	0.5	6.0	0.0	0.3	39.3	2.5	0.0
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
2	2022-10-31 23:30:42	2022-10-31 23:40:16	null	1.7	null	null	158	90	0	9.95	0.0	0.5	2.93	0.0	0.3	16.18	null	null
1	2022-10-31 23:42:05	2022-11-01 00:13:58	null	17.5	null	null	132	68	0	52.0	1.25	0.5	9.46	6.55	0.3	72.56	null	null
2	2022-10-31 23:52:34	2022-11-01 00:23:32	null	12.6	null	null	116	37	0	40.67	0.0	0.5	5.26	6.55	0.3	53.28	null	null
2	2022-10-31 23:16:12	2022-10-31 23:32:36	null	6.23	null	null	158	166	0	22.5	0.0	0.5	5.67	0.0	0.3	31.47	null	null
2	2022-10-31 23:15:00	2022-10-31 23:20:00	null	0.72	null	null	142	142	0	9.95	0.0	0.5	2.92	0.0	0.3	16.17	null	null
1	2022-10-31 23:20:21	2022-10-31 23:34:15	null	2.7	null	null	163	68	0	12.0	0.5	0.5	3.95	0.0	0.3	19.75	null	null
2	2022-10-31 23:45:37	2022-11-01 00:00:39	null	2.45	null	null	249	162	0	11.97	0.0	0.5	3.39	0.0	0.3	18.66	null	null
2	2022-10-31 23:56:35	2022-11-01 00:10:11	null	2.25	null	null	137	50	0	12.68	0.0	0.5	3.16	0.0	0.3	19.14	null	null
2	2022-10-31 23:22:00	2022-10-31 23:28:00	null	1.22	null	null	142	161	0	10.0	0.0	0.5	1.0	0.0	0.3	14.3	null	null
2	2022-10-31 23:25:00	2022-10-31 23:48:00	null	4.7	null	null	186	45	0	19.07	0.0	0.5	4.97	0.0	0.3	27.34	null	null
2	2022-10-31 23:32:54	2022-10-31 23:33:02	null	0.0	null	null	264	263	0	16.82	0.0	0.5	4.58	0.0	0.3	22.2	null	null
1	2022-10-31 23:21:34	2022-10-31 23:32:42	null	0.0	null	null	166	116	0	8.93	0.0	0.5	0.0	0.0	0.3	9.73	null	null

How many rows of data did we get rid of?

data.shape[0] - clean_data.shape[0]

clean_data.describe()

shape: (7, 20)

describe	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	tolls_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
str	f64	str	str	f64	f64	f64	str	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64	f64
"count"	3.630123e6	"3630123"	"3630123"	3.630123e6	3.630123e6	3.630123e6	"3630123"	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6	3.630123e6
"null_count"	0.0	"0"	"0"	126893.0	0.0	126893.0	"126893"	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	0.0	126893.0	126893.0
"mean"	1.712003	null	null	1.384814	3.554337	1.413354	null	165.373444	162.575655	1.174523	15.200393	1.001434	0.494995	2.854254	0.571016	0.299767	22.190213	2.319343	0.103112
"std"	0.45283	null	null	0.931426	4.456602	5.684473	null	65.14861	70.034565	0.504213	13.14293	1.242161	0.0556	3.227445	2.036301	0.009334	16.768073	0.648033	0.343979
"min"	1.0	"2022-09-30 14:06:36.000000000"	"2022-09-30 14:23:04.000000000"	0.0	0.0	1.0	"N"	1.0	1.0	0.0	0.0	-22.18	-0.5	0.0	0.0	-0.3	-4.55	-2.5	-1.25
"max"	2.0	"2022-11-01 01:27:35.000000000"	"2022-11-03 17:26:46.000000000"	9.0	462.86	99.0	"Y"	264.0	264.0	4.0	950.0	10.8	25.48	500.0	516.75	0.3	950.3	2.5	1.25
"median"	2.0	null	null	1.0	1.9	1.0	null	162.0	162.0	1.0	10.5	0.5	0.5	2.25	0.0	0.3	16.55	2.5	0.0

That's looking better - we still have some negatives and other strange looking things, but you get a sense of how we'd approach getting rid of those. For next steps I'd probably filter more things that look incorrect, then I'd plot the data as a histogram and eyeball it to see if there's anything strange in there.

Let's stop there.

This was my first foray into Polars, so it's likely there are more polarsy ways of doing things - if you know of improvements please let me know.

Overall I really like Polars. It's fast and clean and the resulting code is easy on the eyes. I'm likely to use it in place of Pandas going forward.

Links

Modern Polars is quite nice with many examples.

Towards Data Science has a good article called Polars: Pandas DataFrame but Much Faster.

This article from Code Magazine has a lot of good Polars examples.

The Polars documentation page on Coming from Pandas is very useful if you have Pandas experience.

While we're at it, read this piece from a former cab driver on the ride he'll never forget.

Blog Technical Setup

2023-02-16T23:33:00-08:00

Overview

I'm using Pelican as the static website creation framework.

I created the theme myself, I can open sores it if anyone is interested. It uses tailwind, which is mostly quite good, but a pain when it comes to integration with Pelican: tailwind wants classes added to html, but the html is generated (as opposed to hand-written), so it's hard to style.

The website is being served via github pages.

I wrote a custom script to create redirects from the old wordpress style URLs to the URLs.

Authoring: I write in markdown, run Pelican to generate the html, commit the output in git, push to github, and the website is updated. Fairly clunky, I'd like to make that smoother.

Everything Sad is Untrue: Book Review

2022-12-26T23:38:00-08:00

Author: Daniel Nayeri

Quite good, nostalgic, I immediately decided to look for other books by the author.

Parable of the Sower: Book Review

2022-12-26T23:38:00-08:00

Post-apocalyptic non-sci-fi book. Entertaining, not a bad read, not amazing. Verdict: Maybe Read It

Banshees of Inisherin: Review

2022-12-26T23:33:00-08:00

Verdict: See it

Entertaining, quite funny, unexpected turn.

"Python itertools.groupby : a clearer example"

2013-01-16T10:00:00-08:00

itertools.groupby allows you to group a sorted list (or any sorted iterable) by some key, giving you the grouping key and the list of items grouped by that key.

For example, say you have a Django the model:

class Item(models.Model):
    user     = models.ForeignKey( User, verbose_name="User", db_index=True )
    amount   = models.DecimalField( max_digits=19, decimal_places=2, default=0, null=True, blank=True )

And you want to group your Items by User. It'd be something like this:

ordered_items = Item.objects.all().order_by('user')
for user, item_list in itertools.groupby( ordered_items, lambda item: item.user ):
    print user, list(item_list)

That'll display the user, followed by all of the Items for that user, for each user.

Now let's say you wanted to display the user and the total amount for all of the items. You'd do something like this:

ordered_items = Item.objects.all().order_by('user')
for user, item_list in itertools.groupby( ordered_items, lambda item: item.user ):
    print user, sum( item.amount for item in item_list)

"Undo last git commit / merge"

2012-07-27T10:00:00-07:00

Note to self: I quite stupidly merged and committed an unwanted branch into master. Took a surprisingly long time to figure out how to get rid of it:

git reset --hard HEAD^

Which means reset git to one commit before the current head, effectively getting rid of the last commit.

Also, to undo a merge into master, you can reset back to origin/master with:

git reset --hard origin/master

More scenarios for undoing merges on this stackoverflow page.

"Efficiently finding large (size and dimension) images"

2012-05-09T10:00:00-07:00

I needed to efficiently find large images (those with dimensions greater than x). Here's what I came up with:

First, find images whose file size is greater than 3M:

find . -name '*jpg' -size 3M > /tmp/big-receipts.txt

Now, get the dimensions of each of those images using imagemagick's identify command line utility. The key to using this thing efficiently is to include the "-ping" flag (believe it or not that took me quite a while to figure out):

identify -ping filename.jpg

displays this type of output:

/tmp/big.jpg JPEG 10200x13200 10200x13200+0+0 8-bit DirectClass 5.54MB 0.000u 0:00.000

Now we can put the two together:

for fname in `cat /tmp/big-receipts.txt`; do identify -ping $fname; done

"Hands On With the BlackBerry Playbook: Actually Quite Nice"

2012-02-27T10:00:00-08:00

Over the weekend I attended the Blackberry hackathon with the hopes of creating an Xpenser blackberry app. The RIM folks were nice enough to give away a Playbook to everyone who presented, so I ended up with one.

I was skeptical, to say the least. I expected a sluggish, ugly interface with a dearth of applications.

What I got is quite nice. The blackberry has a very usable interface - it's fast and responsive, and some of the gestures (eg. closing and switching apps) are the best I've seen on a tablet.

It comes with a nice car racing game (I surprised myself by spending a few hours playing it), and the browser is quite excellent, probably the best I've seen on a tablet.

The development environment is also rich: you can use HTML or flash as well as native SDKs.

It doesn't seem to come with a built-in email app (or if it does I've missed it), which is very surprising given the Blackberry pedigree.

It's also lacking apps and games - the app world feels a bit empty.

I'm surprised at how much I like the device. I just wish it had more apps.

"SFTP: Specifying Port, SSH key"

2012-02-23T10:00:00-08:00

This page is helpful.

Specifying port:

sftp  -v -oPort=6636 -o IdentityFile=keys/my-key.ssh [[email protected]](/web/20120914064722/http://cloudflare.com/email-protection.html)

"Student Loans Are Killing Entrepreneurship"

2012-02-15T10:00:00-08:00

If you know me you know I'm a huge fan of entrepreneurship - instead of renting yourself out to someone else, start a business that scales. Be your own pimp.

I gave a guest lecture at UCSD recently espousing this viewpoint. You are cheap right now, I told them, so take advantage. If you wait 10 years to start a company you'll have much higher earning power and much higher personal costs (spouse, mortgage, …) and therefore a much higher opportunity cost when starting a company.

What about our student loans? asked one of the students.

Good question. I don't have a good answer.

When I went to school I was able to pay my way from the third year onward. I left school with savings instead of debt. I was able to start a company immediately.

Today students graduate with an average debt of about $27,000 dollars.

That's just awful. That means they are bound to a job for at least the first part of their career. We're removing the option of starting a company, of producing explosive value, of growing the economy to the benefit of the whole country, for more than 60% of our graduates.

Think of all of the companies with billions in value created by college students. Think Facebook, Dropbox. We need these. We want these created in the US.

We have to get education costs under control. We can screw the young generation over to put a few extra bucks in our pockets now, but we'll all end up paying for it when we fall behind other countries who encourage their creative young to be entrepreneurs.

"How To Dynamically Import Python Modules"

2012-01-11T10:00:00-08:00

Here's how you import a python module dynamically: say your module is called "mysettings" and it's in the directory "config". Make sure you have a __init__.py in your "config" directory and use:

mysettings = __import__("config.mysettings", fromlist=["config"])

This is equivalent to

from config import mysetings

"How To Scan Multiple Pages Into a Single PDF on OS X"

2011-11-15T10:00:00-08:00

My new printer has a document feeder which makes scanning of multiple pages easy, but I didn't know how to kick off the scan on OS X. Turns out it's quite easy:

Open the OS X Preview app
Go to the File menu and select Import From Scanner
Select your printer
Look for the "Format" option on the right hand page and set it to PDF
Select the "Create single document" option
Hit the Scan button

This will produce a single PDF file with all of the pages of your document, scanned into the location of your choice.

"Titanium Studio: How To View Android Log Messages"

2011-08-15T10:00:00-07:00

Apparently with Titanium Studio it's not possible to see log messages in the Titanium console.

Here's how you see your log messages:

adb logcat | grep "TiAPI"

adb is in the tools directory of your Android sdk.

"Easy way to view cookies in Chrome"

2011-07-27T10:00:00-07:00

Here's an easy way to see the cookies for the current site in Chrome:

javascript:void(document.cookie=prompt(document.cookie,document.cookie));

Via stackoveflow.

"Uniforms"

2011-03-04T10:00:00-08:00

I went to a school that required uniforms (private school in England). We were forced to be uniform.

This afternoon driving by the high school near my house I was amused to see uniforms everywhere - here a patch of girls all wearing the same types of shorts, with hair pulled back, there a group of emo kids all wearing shades of near identical clothes.

It's lovely. We all need to be different in our uniform ways.

"Future of Work and New Dial Tones"

2010-12-08T10:00:00-08:00

My friend Paul Kedrosky's post on one of his main investment theses, the application of currently consumer oriented technologies to business (DropBox, Twitter, FaceBook, Yelp, …), got me thinking about how the future of work will look.

I've made a career of working far from where I live. I've generally considered this a significant pain - there are disadvantages and inefficiencies in being geographically separated from coworkers. Hence I've endured horrible commutes and frequent travel.

Lately, however, I've been reconsidering. My office is about 30 minutes from my house, but I end up not going in more and more these days. The 1 hour roundtrip is simply wasted time, and the traditional 9-5 is an artificial constraint on both work and non-work activities.

The turning point, not surprisingly, was technology driven - earlier this year I was working on a shared presentation and document with 2 colleagues. We met in person twice, and the third time we used Google Wave (rip) for shared document editing and iChat for voice and video communication.

I was shocked at how much more efficient the third session was. I despise video calls, but in this case it was background, mainly to see who was paying attention and who was tuned out. The shared document with real-time edits worked fabulously well - one could write stream of consciousness thoughts, and before finishing the paragraph have his work edited and polished by the other two.

These days I find my most efficient working relationships require little in-person interaction. What they do require is a host of enabling technologies we take for granted.

I'm still a huge believer in the value of in-person interactions - the best ideas generally pop up over lunch or other informal settings. But in-person interaction doesn't require the traditional office setting and trappings.

The work style adopted by startups today will make its way to large business, and the services that are the lifeline of that style will become the dial-tone of business. As Paul points out this will drive the creation of a number of significant companies to provide those dial-tones. And with any luck I'll have an opportunity to invest if a few of them

"Generating static html from markdown in python"

2010-11-29T10:00:00-08:00

Note to self, for future reference.

Markdown reference

Generating static html:

import markdown2
file('classification.html','w').write( markdown2.markdown_path('classification.md', extras=["code-friendly"]) )

"Retention Bonuses Don’t Work"

2010-11-19T10:00:00-08:00

Retention bonuses don't work because they only motivate presence, not contribution.

"mo.com Interviews Me"

2010-11-08T10:00:00-08:00

The lovely folks at mo.com recently asked me about my experiences at Yahoo, the background and business model for Xpenser, and thoughts on funding and raising VC money. The interview has been published here, drop by and take a gander.

"How To Run A Command On Multiple Linux Boxes"

2010-10-29T10:00:00-07:00

Here's a handy way of running a command on multiple boxes. In this case I wanted to check disk space on a series of machines.

Assuming you're using bash as your shell, the following will list the hostname and run df on machines racb13 through racb28 and spit out the results.

for i in {13..28}; do ssh racb$i 'echo `hostname`; df -h | grep /data'; done

Note that ssh has to be setup to allow password-less login and so forth, but I'm assuming you're smart enough to know how to do that or can look it up.

"Art"

2010-07-23T10:00:00-07:00

I don't know why but I'm seriously in love with this drawing from the six year old; it seems to have a real style. Better than anything I could draw, methinks.

"How To Be A Good Participant On A Panel: Disagree"

2010-03-04T10:00:00-08:00

Mark Suster and Fred Wilson have both posted on the topic of being a good panelist. I'll throw in the best advice I ever got on the topic (from Alex, who got it from someone else):

Disagree with other panelists.

A panel of people agreeing with each other is generally tremendously boring, and any topic that's interesting enough to be brought up as a question to panel must have at least two sides that can be argued.

So argue the other side. Find someone else on the panel willing to go toe to toe with you, and disagree with what they say. The point is not to start a fight, but to create an engaging discussion. It'll force you to think on your feet and it'll entertain the audience.

"Django: Using The Permission System"

2010-02-19T10:00:00-08:00

I was surprised at how little information I found on making use of Django's permission system. Here are some quick notes on one way to use it:

Groups are groups of users. For example, you could define a group of users who have premium accounts, or have been verified in some way, or are somehow special:

from django.contrib.auth.models import Group, Permission
special_users = Group(name='Special Users')
special_users.save()
really_special_users = Group(name='Super Special Users')
really_special_users.save()

Now you have two groups defined and can define permissions for them. Django associates permissions with models (note: not model instances, but models). You'll need to select a model to apply the permissions to, and do a small dance with "ContentType" to find that model's content type:

from django.contrib.contenttypes.models import ContentType
somemodel_ct = ContentType.objects.get(app_label='myapp', model='somemodel')

can_view = Permission(name='Can View', codename='can_view_something',
                       content_type=somemodel_ct)
can_view.save()

can_modify = Permission(name='Can Modify', codename='can_modify_something',
                       content_type=somemodel_ct)
can_modify.save()

You've now defined two permissions and can associate them with your Groups:

special_users.permissions.add(can_view)
really_special_users.permissions = [can_view, can_modify]

Our groups and their associated permissions are ready to go. Now we just have to associate these permissions with users:

jack=User.objects.get(email='[[email protected]](/web/20120325165629/http://cloudflare.com/email-protection.html)')
jack.groups.add(special_users)

jill=User.objects.get(email='[[email protected]](/web/20120325165629/http://cloudflare.com/email-protection.html)')
jill.groups.add(really_special_users)

We're all done. Now we can check the users' permissions:

>>> jack.has_perm('myapp.can_view_something')
True
>>> jack.has_perm('myapp.can_modify_something')
False

>>> jill.has_perm('myapp.can_view_something')
True
>>> jill.has_perm('myapp.can_modify_something')
True

And to use it in your templates:

{% if perms.myapp.can_view_something %}
Here is something for you to see.
{% else %}
Can't show you!
{% endif %}

"Hyper Competitive Sleep Losing Entrepreneurs?"

2010-01-31T10:00:00-08:00

I remarked to a friend that Mark Suster's entrepreneurial roots show in his approach to being a VC - he's come out of nowhere and in short order aggressively pushed himself into being one of the most relevant voices out there. He seems to be working a lot harder than the others guys, exactly as you'd expect an entrepreneur to be. Not exactly what you'd expect of a VC.

His blog is fantastic, and I quite often agree with his advice.

That's why I found it odd that his The Best Entrepreneurs Are Hyper Competitive & Hate Losing struck such a dissonant chord with me.

Shuffling through the successful business people I know and trying to gauge whether they would be the type of people who are obsessed with winning, even in a family game of scrabble, I don't come to a clear conclusion. I know hyper competitive people, but I also know plenty of people who are able to separate their business behavior from their personal behavior. And not obsess with beating the competition.

Maybe that's what's not sitting well with me - Mark's definition of winning seems to be beating the competition.

Some of the best entrepreneurs I know don't obsess with the competition. They obsess with their own behavior.

Here's a contrived example - look at Apple. Do you see Jobs competing with the others in the industries he enters, or do you see him trying create the best possible product, distinctly separate from what his competitors are doing?

Frankly I have a hard time picturing a lot of these guys stressing out over scrabble or Guitar Hero.

Mark's a very successful guy and his approach has certainly worked for him, but I disagree that you need to be obsessed with winning in the way that Mark describes it.

Look at this way: you could destroy all your competitors and still not win. You could also win without destroying any of your competitors.

First, pick the right game. Then, pay attention to playing that game as best it can be played. Competing may be an important tactical part of playing the game, but it's probably not the part to obsess over.

"Django-mptt: Tree Storage in Django: A Brief Overview"

2010-01-24T10:00:00-08:00

django-mptt is a library for storing tree oriented data using the Django ORM. It allows you to place your model instances into a tree structure and efficiently query for ancestors and children.

Here's a brief tutorial on how to use it:

After installing, you'll need to modify your model to include a "parent" field, and register it with mptt:

class Person(models.Model):
    contact   = models.ForeignKey( Contact, db_index=True )
    role      = models.CharField(max_length=20, blank=True)
    parent    = models.ForeignKey('self', null=True, blank=True, related_name='children')

    def __unicode__(self):
        return "Person: <%s>" % (self.contact.email, )

mptt.register(Person)

mptt dynamically adds fields to your model, so you'll need to syncdb after you've added the parent attribute and the mptt.register call to your model.

The basics are fairly easy to use:

To move a node to the root of the tree, use move_to with a targe of None:

person1.move_to(None)
person1.save()

To make a node the child of another, set its parent:

person2.parent = person1
person2.save()

To find the children of a node, use the children field:

>>>person1.children.all()
[<Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>, <Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>]

Here's a little snippet of code to setup a 15 node tree where each node has two child nodes:

[UPDATE] The code in this snippet is not correct - you have to save each node as you update it, then look it up again. You can't modify a node, save it, then use the reference you already have for it. I'll update the code when I get a chance

contacts = []
people = []
for n in range(15):
    c = mod.Contact(email="test" + str(n) + "@testing.com")
    c.save()
    contacts.append(c)
    p = mod.Person(contact=c)
    p.save()
    people.append(p)

people[0].move_to(None)  # Root
people[0].save()
for n in range(1,15):
    people[n].parent = people[(n-1)/2]
    people[n].save()

Now let's take a look around:

>>>people[7].parent
<Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>

>>>people[3].children.all()
[<Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>, <Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>]

Now let's move things around a bit; we'll take person3, which is 2 levels down from the root, and make it a direct child of the root:

>>>people[3].parent = people[0]
>>>people[3].save()

>>>people[0].children.all()
[<Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>> <Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>, <Person: Person: <[[email protected]](/web/20120414080551/http://cloudflare.com/email-protection.html)>>]

And we can look at the ancestors of a given node:

people[14].get_ancestors()

Fantastic Mr. Fox: See It

2010-01-23T10:00:00-08:00

I'm a big fan of Fantastic Mr. Fox. We just watched it again and it was a hit with everyone from the 2 year old to grandma.

The dialog an voice acting are excellent, and the animation is really refreshing. The story is smart and is different enough from your typical Disney fare to be a welcome change.

I recommend it, go see it.

"Feels Like A New Stage Of The Web"

2010-01-09T10:00:00-08:00

I was doing some log analysis this morning and was struck by the variety of user agents accessing Xpenser. Quite a bit of mobile access with quite a variety of different browsers, as well as some more exotic items (various tablet PCs I don't recognize, etc).

Then there's the API access that's been picking up quite significantly - in fact, I wouldn't be surprised if it overtakes regular web access in a little while.

For quite a while the web we had a fairly uniform set of entry points - IE and FireFox for the most part, with a smattering of others thrown in. There is so much more variety now, and the variety is significant - for example, the difference between a mobile device and a desktop browser is much more significant than between IE and FF.

I'm hoping other modes of access become commonplace as well - boxee, tablet PCs, etc.

It feels like we're at a transition point from uniformity to diversity again. This will mean pain the short term as we adjust, but a richer and more encompassing experience once we make it to the other side.

I'm looking forward to it.

"Move Files Older Than X Days To Another Directory"

2010-01-08T10:00:00-08:00

Here's a little script for finding files modified more than 7 days ago and moving them to another directory:

find . -type f -mtime +7 -print > /tmp/old_files.txt
cat /tmp/old_files.txt | while read line; do mv "$line" ../old_files ; done

"How To List Files In A Package in Ubuntu"

2009-11-15T10:00:00-08:00

dpkg -L package-name

"How To Enable Vi Syntax Highlighting In Ubuntu"

2009-10-28T10:00:00-07:00

Note to self, as I seem to need to do this on every new install:

Ubuntu ships with vim-tiny, which doesn't support syntax highlighting. Do this:

sudo apt-get remove vim-tiny
sudo apt-get install vim
sudo vi /etc/vim/vimrc
( Remove the quote mark from the "syntax on" line, uncommenting it …

"The Power of Stories In Shaping Views of Cultures"

2009-10-17T10:00:00-07:00

Impressive Ted Talk by Chimamanda Adichie on how stories, and in particular "single stories", shape our views of cultures. Many parallels with how people typically see Iran.

A great quote from the talk: "The problem with stereotypes is not that they are untrue - it is that they are incomplete".

"Eclipse + PyDev : I Recommend It"

2009-10-11T10:00:00-07:00

I used to be a vi guy who finally made the move to graphical editors. I looked for the simplest, lightest possible solutions, using ConTEXT for quite a while.

Some years ago I was forced into using Eclipse for reasons I can't quite recall; probably Java development. I didn't like it - the forced Project concept, the bloat, the general slowness.

Eventually I got comfortable with it, got PyDev installed, and made it my primary development environment. These days most of my development lives in Eclipse.

With the 1.5 release PyDev included quite a few previously pay-only features in the free / open source version. Since then I've found I've become even more productive in the environment, and now actually enjoy it.

In particular, the code analysis is very useful. I love the fact that it points out unused imports and variables as well as syntax errors. Going through old code I was surprised at how many spurious imports I had, as well as a few actual errors in code that had been in production for several years in rarely exercised branches.

If you're doing python development I recommend you take a look at Eclipse+PyDev. I was surprised at the level of increased productivity it brought me.

"How Interviewing Job Applicants Has Changed"

2009-10-11T10:00:00-07:00

I'm noticing how much the practice of finding a candidate and interviewing them has changed over the years.

First step is to look up the person on the web. Since I'm looking for a technical person I'm expecting a blog, contributions to open source projects, a twitter presence, etc. If I don't find them, the candidate becomes less attractive.

If I do find a blog or twitter feed, I take some time to read them. You can tell a lot about a person with just a short overview - for example, even a delicious feed can give you a very good feel for how good someone is technically.

Hopefully the person has an open source project or two they've contributed. If so you can dive right into the code and get a good sense of their skills at work. I love this - you can see great coders almost immediately. And of course bad ones.

If the person is still looking good it's time for a review of the resume and their linkedin profile.

By the time I followup with the person I have a very good sense of who this person is, what they've done in the past, and what their real-world coding looks like. The people who make it to the interview are so well filtered you almost can't go wrong.

It's very time consuming, but if you're as fanatical about hiring good people as I am it's worth it.

Contrast this with the old pile-o-resumes approach, or the recruiter-filter-by-keyword approach, and I think we're in a much better place.

Now if I could only find a recruiter I could trust to do the technical deep dive I've described here and be ingenious enough to find diamonds in the rough I'd be a very happy man.

"Yahoo Mail WTF?!"

2009-09-28T10:00:00-07:00

I still have a big soft spot in my heart for Yahoo, but Yahoo mail, WTF?

Of the entire screen only the area I've highlighted in orange is dedicated to the contents of the message. And this is with the app maximized to full screen.

The interface is unusable as a mail reader: I can't read the email message.

How on earth did the product people agree to this? Where's the customer advocate?

"What Should I Read Next?"

2009-02-16T10:00:00-08:00

I was thinking about what book to read next and it occurred to me to ask you nice people. I'm in the mood for something factual but interesting - something like The Prize, if you're familiar with that. I was intrigued by Fiasco but I haven't looked into it.

So what do you say, what should I read next?

"Another Python Convert"

2009-02-13T10:00:00-08:00

Looks like I've finally converted Alex to Python as well. "This is actually shorter than perl and still makes sense". Be forewarned - if you code with me, you'll be doing Python in no time.

"Ctrl-Q, My New Favorite Eclipse Hotkey"

2009-02-13T10:00:00-08:00

Due to aforementioned OS X stupidity with the Home and End keys, my new favorite Eclipse hotkey is Ctrl-Q. It takes you to the location of your last edit. So next time you hit the End key to go to the end of the line but end up at the end of the file and curse Jobs and his cult, remember to hit Ctrl-Q to go back to where you should be.

"San Jose San Fran Next Week, Drop Me A Line If You Want To Meet"

2009-02-03T10:00:00-08:00

The title pretty much says it; I'm planning an SJC/SF trip next week, if you want to meet or chat drop me a line - Darugar at gmail.

"On Concurrency, Threads, and Python"

2009-02-01T10:00:00-08:00

Nice and detailed article from Jesse Noller on concurrency, threads, the GIL, and Python. Worth a read; I learned a lot from it.

For the record I'm one of the people who doesn't think threads are not the true solution to the concurrency problem.

Books Cannot Be Owned

2009-01-25T10:00:00-08:00

Wikihow's How To Stop Accumulating Books has been making the rounds in blogs and delicious popular. Reminds me of the ultimate book non-accumulator, a good friend of my dad's.

This particular friend of my dad's would always bring me books when he came to visit. I was a voracious reader, to the point where people made fun of me for it, so I always loved his visits. And he brought interesting books - not children's books, but histories, etc. I think he brought me Animal Farm when I was 8 or 9.

As I got older I started to notice something odd about the books - they didn't belong to him.

Eventually I found out he didn't own any books. He'd take a book from you, read it, and give it to someone else. Usually without permission.

I was fascinated. I remember asking him about it, and he told me: books cannot be owned. Ideas are the ultimate expression of freedom, and to cage them is unnatural. Books are for reading and for passing on.

That made a big impression on me. I've always thought sharing books with friends is the most amazing gift.

Anyway, that's why I've never returned that book you lent me

"Facebook: Nice Place To Visit, But I Wouldn’t Want To Live There"

2009-01-25T10:00:00-08:00

Yet another Scoble post on a facebook accounts being closed, this time with the iFart app author.

Facebook is like Disney Land. It's a nice and enjoyable place, but it has its own special governance. It can decide when to let you in and when to throw you out.

It's a great place to visit, but I wouldn't want to build my house there. Not when the lease can be revoked at any time.

All my data is belong to me. It makes no sense to put a third party in control of it. My photos: mine. My writings: mine. Why would I hand them over to a third party?

I can understand the appeal of easy publishing to the general public - Facebook truly makes it easy for people to establish a presence and communicate. But for the more technically savvy, Facebook should be treated as a copy, not as the source. Data should be originated and owned elsewhere and channeled into Facebook. Write in your own blog. Post photos onto Flickr, with a local and web backup strategy. Funnel them into Facebook.

That's a bit more like FriendFeed, which may be why Scoble likes it so much.

"Hackintosh Accomplished"

2009-01-25T10:00:00-08:00

After 2 days of pain, finally a hackintosh is born. My advice to you if you're thinking of setting up a hackintosh: don't. Go buy a cheap mac instead.

As I was fond of saying in the old days: Linux is free if your time is worthless. Hackintosh is about the same.

Having said that, I can confirm OS X 10.5.6 runs on a Toshiba A135-S4467 with most things working, with the exception of: sleep and wifi (so far). Things that do work: wired ethernet, USB, both external keyboard/mouse (with USB) and internal laptop keyboard/mouse, sound, and dual boot with Vista.

I'll probably post the painful details at some point. In the meanwhile, I'm back on the mac. The things one does for the sake of the iPhone…

"Protect The Constitution Of The United States"

2009-01-20T10:00:00-08:00

Inaugration day, and I missed just about all the festivities. One thing did stand out for me: the text of the Oath of the Office of the President:

"I do solemnly swear that I will faithfully execute the Office of President of the United States, and will to the best of my Ability, preserve, protect and defend the Constitution of the United States."

Preserve, protect, and defend the Constitution of the United States. Those founding fathers were smart.

Here's hoping the new president takes the constitution more seriously than the last one and defends it instead of trying to diminish it.

"Queueing Benefits"

2009-01-19T10:00:00-08:00

Queues are nice things. We should be using more of them.

A couple of benefits that I knew of but didn't really appreciate until Alex started using beanstalkd in his application:

- Having a worker-pulls-jobs-from-queue model provides near optimal use of the machine and prevents overload and thrashing. Setup as many simultaneous workers as your resources can handle and let them go. You have a controlled number of workers, preventing thrashing, and your workers work continuously. You don't have to worry about a load distribution strategy - workers pull jobs as fast as they can.

- Provisioning new workers into the system becomes trivial. Want to add another box into the mix? Just set it up and have the workers start pulling jobs. You don't have to worry about registering the new box and getting it into the load distribution system - so long as it knows how to connect to the queue it can grab jobs. Alex commented that scaling his system is as easy as bringing up another virtual machine - as soon as it's up it starts pulling jobs.

These are both very important operational benefits that I had largely ignored.

Parandism: Two Man Job

2009-01-13T10:00:00-08:00

Parand says:

When attempting to do a two man job with only one man, substitute endurance for raw power.

"San Diego Hadoop User Group Starting"

2009-01-04T10:00:00-08:00

We're starting a San Diego Hadoop User Group. Please send me an email ( darugar at gmail ) if you're interested or leave a comment here. Details to be worked out, but we'll likely will have the first meeting towards the end of February.

"Obama Logo Design"

2008-12-14T10:00:00-08:00

Interesting video of how Obama's campaign logo was designed. Via LogoDesignLove.

"Cloud Computing Hadoop Slides"

2008-11-20T10:00:00-08:00

Slides from my Data Processing in the Cloud talk:

Cloud Computing: Hadoop

View SlideShare presentation or Upload your own. (tags: pig cloud)

"Blogs Are Becoming Traditional Magazines"

2008-11-18T10:00:00-08:00

Have you noticed how more and more the headlines of the top blogs are all the same announcement at the same time? Unfortunately we're going back to the PR-firm driven news reporting that was prevalent in the era of magazines, instead of that brief but wonderful time when bloggers actually thought for themselves and found stories of their own liking. At least magazines had standards related to grammar and some base level of fact checking, but oh well.

On the bright side the blog world is wide and deep enough to provide lots of interesting news sources, even if they turn out not to be the top blogs.

"Python Dot Notation Dictionary Access"

2008-10-24T10:00:00-07:00

In most cases I prefer dot notation over bracket notation for dictionary access. That is, I prefer mydictionary.myfield over mydictionary\['myflield'\]. I also prefer attempted access to undefined keys to return None instead of raising an exception.

With the help of this thread, this is what I've been using:

class dotdict(dict):
    def __getattr__(self, attr):
        return self.get(attr, None)
    __setattr__= dict.__setitem__
    __delattr__= dict.__delitem__

>>> dd = dotdict()
>>> dd.a
>>> dd.a = 'one'
>>> dd.a
'one'
>>> dd.keys()
['a']

>>> existing = {'a':'A', 'b':'B'}
>>> dot_existing = dotdict(existing)
>>> dot_existing.a
'A'

"Python Scripts For Dumping Oracle Data And Loading Onto Hadoop DFS"

2008-10-22T10:00:00-07:00

There have been several requests for this, so I might as well post it here for general use. I put together a simple system for dumping data out of Oracle databases and loading onto Hadoop DFS. The slightly interesting part is the parallelism - Python's Processing library is used to dump partitions in parallel and copy and load them onto DFS in parallel. This helps when dumping large amounts of data from partitioned Oracle tables.

The database interaction is handled by db.py . There are a couple of helper functions for finding table partitions, etc. DBDumper dumps the requested fields from the requested table:

dumper = db.DBDumper('username/password@yourhost:9999/DB', 'table_name',
      ('field1', 'field2', 'field3'), 'owner', 'partition', 'output_dir', 10)
dumper.dump(cp)

Where 10 is the level of concurrency, owner is the owner of the table, and partition is the name of the partitions you're interested in (can be None).

dfs.py copies the dumped files over in parallel, again using PyProcessing. It's simply a wrapper around "cat | ssh | hadoop dfs -put".

DBDumper and dfs are tied together via a callback - when each partition is dumped, the callback is invoked, triggering the dfs copy.

Here's a complete example of using these to dump and copy data:

import db
import dfs

fs = dfs.RemoteDFS('address.of.remote.machine')

def cp(arg):
    print "CALLBACK:", arg
    fs.cp(arg[0], '/some/directory/' + arg[1] + '/' + arg[2])

dumper = db.DBDumper('username/password@yourhost:9999/DB', 'table_name',
     ('field1', 'field2', 'field3'), 'owner', 'partition', 'output_dir', 10)
dumper.dump(cp)

"GMail Contact Groups: Add By Tag / Search"

2008-10-20T10:00:00-07:00

I'm trying to setup ad-hoc small mailing lists using GMail. They have a contact groups feature that mostly does what I need. However, the interface for adding contacts is quite limited - you pick from a list.

It'd make a lot more sense to allow addition of contacts to groups based on searching - eg. look for everyone labeled "tester" and add them to the "tester" contact group en-masse, or for any message containing the term "paintball" and add all the senders/recipients to "paintballers" group.

So there you go GMail group, please get to work.

"RESTful URL Design For Search And Collections"

2008-10-16T10:00:00-07:00

I'm trying to find the appropriate design for RESTful design of URLs for search and for collections of items.

The setup: we have two models, Cars and Garages, where Cars can be in Garages. Base URLs:

/car/xxx           (xxx = car id)
/garage/yyy     (yyy = garage id)

Now we want to provide a search for cars - eg. show me all the blue sedans with 4 doors. What's the appropriate URL?

1  - /cars/color/blue/type/sedan/doors/4
2  - /cars/color:blue/type:sedan/doors:4
3  - /cars/?color=blue&type=sedan&doors=4
4  - /car/search/...

None of these are satisfying.

1 through 3 use "cars" as the base (as opposed to "car"). So the pattern for doing searches / collections would be to pluralize the model. This seems ok.

1 has arbitrary ordering of the fields and no good way to distinguish fields versus their values. 2 is slightly better, but still doesn't seem right.

3 uses the QUERYSTRING for the parameters instead of the PATHINFO, and frankly looks better to me, but I've heard of objections to using QUERYSTRING. The problem I have with it is it's not consistent - if I was searching on a single field my URL would probably be: /cars/color/red or something like that. Having the URL drastically change form just because there are more search parameters seems wrong.

4 uses the "car" base url along with the verb "search". That seems wrong - verbs shouldn't be part of the URL, right? It's been suggested several times though.

Now a slightly different case - let's find all the cars in a given garage:

1  - /garage/yyy/cars
2  - /cars/?garage=yyy

1 seems pretty good in this case.

Please chime in with your thoughts, either in the comments here or in the Stackoverflow thread.

"Yahoo Live Surprisingly Good"

2008-10-16T10:00:00-07:00

I'm using Yahoo Live to get a look at the Yahoo UCSD Hack Day event, and I'm quite surprised at how well it works.

Seeing someone's face via their webcam is an absolute waste of time, particularly on on video conference calls, but here I'm looking at a view of the room in the larger window, and at several of the hackers in smaller windows. This is actually a pretty workable virtual office. I can see Paul eating Pizza, Rasmus helping people, and the group I was sitting with mostly still there. I half feel like driving back over and re-joining the group.

Why is this useful? Well, for one, I can see people working, and it motivates me to stay up and work. Compare this to having no communication, just email, or IRC, and you'll see it provides a much more tactile, real-world feel. I can see and hear the activity.

I could actually imagine a technology like this being useful for virtual office. Kind of like hanging out on IRC/IM, but with a more human feel.

"Fremium Works If You Do It Right"

2008-10-15T10:00:00-07:00

I am an ~~incredibly cheap bastard~~ frugal. I don't like to pay for things.

Yet I just paid for Flickr without hesitation or a moment of doubt. Odd.

Here's why, I think:

Cheap: $25/year is not a lot of money.
Excellent service. I've never had a problem with the service; they're always available and fast.
Nice, easy to use product.
Trust. I trust that Flickr won't do anything bad with my account, with my photos, with my credit card, or with anything else. I trust them because I believe they respect their customers:
They don't automatically charge renewals your account - you have to take an action each time you have to pay them, so you know they're not going behind your back to charge your credit card.
If you decide not to pay, they still provide a reasonable product (only your most recent 200 photos are available) instead of leaving you high and dry. If you decide to pay at a later time you get your full account back.
They provide full, easy access to all your data via a variety of APIs and tools at all times. They're not trying to tie you down by trapping your data.

I feel comfortable spending money on Flickr in the same way I feel comfortable spending at Costco - I feel they'll make things right for me without giving me a lot of trouble. So I spend.

This is the way to build a sustainable fremium business - make gaining your customer's trust your top priority.

"Python Script For Finding And Removing Duplicate Files"

2008-10-15T10:00:00-07:00

My image, mp3, and ebook collection were a mess after years of copying to various servers, consolidating, and re-copying. I had lots of duplicates.

I looked for an app to find and remove duplicates but surprisingly didn't find anything very good. So I had to write my own.

This is a very simple script - it scans the directory tree you specify, looks for exact duplicates, and removes the duplicates.

It's not very smart about which copy it removes. It's not smart about finding files that are "similar" - it only finds exact matches. It ignores small files (intentionally - it's easy to make it deal with small files).

It uses /temp for its output and cache files, so it's targeting windows. Change that to /tmp if you're running unix.

I built in a caching mechanism to save the results of scanning the disk, but it turned out not to be too useful and the script ran faster than I expected, so the caching is commented out.

Here it is: FileInfo.py .

"Small Biz Minimal CRM Thoughts"

2008-10-15T10:00:00-07:00

As the "computer guy" for my wife, various friends, and my own activities, I often see a need for a very simple "CRM" system. But it's not really CRM; it's more of a contact management system together with a way to send emails to various groups of friends / customers, a way to catalog past exchanges, and perhaps a simple to-do system.

Periodically I look at the various available options (there are many), give one or more of them a try, and give up, generally because they're too complex and too capable.

The most promising is High Rise, but I don't think it provides a way to send emails, so it doesn't qualify.

Here are my requirements for my imaginary Pony CRM:

Simple. Must be simple enough that my wife will actually use it.
Integration with Yahoo Mail / GMail. This should be generalized to POP/IMAP integration, but I find almost everyone uses one of the large providers, either Yahoo or Gmail.
Automatic synchronization with Yahoo/Gmail. There shouldn't be an "import" option - it should always be sync'd automatically. A contact in Gmail should be immediately available in Pony CRM, and vice versa.
Easy cataloging / collection of emails. It'd be good to be able to catalog and tag email exchanges.
Easy mass email. This is the most important feature - without exception, the most common use case is: I want to email a subset of contacts and I want to track who replied, looked at what I sent, etc. For example, all of your clients that have certain investments, or all customers who've requested feature X, all customers who signed up before date X, and so forth, should receive a nice email.
Easy graphical email composition. Everyone wants to send a nicely formated email with graphics that's readable in the majority of email readers, but no one can figure out how. This should be built in to the CRM tool in some way. There should be a way to store the email templates for future use.
History and Audit Trails. You should be able to figure out who you sent what to, when.
Extensibility. APIs for access to the CRM system as well as a way to integrate external feeds/ APIs/ widgets into the CRM system.
Free/Cheap Hosting with Simple Full Export. If it's cheap enough, I can push everyone to pay the monthly fee and get the support issue off of my back. However, there must be a simple full-export option as well as an alternative means of using the exported data - eg. an open source version that could be deployed should the hosted version go out of business. Can't have lock-in if your business depends on the data.

I think this covers just about everything needed. Someone please either build or point me to Pony CRM so I can be happy.

"Access Python Dictionary Keys As Properties"

2008-10-13T10:00:00-07:00

Say you want to access the values if your dictionary via the dot notation instead of the dictionary syntax. That is, you have:

d = {'name':'Joe', 'mood':'grumpy'}

And you want to get at "name" and "mood" via

d.name
d.mood

instead of the usual

d['name']
d['mood']

Why would you want to do this? Maybe you're fond of the Javascript Way. Or you find it more aesthetic. In my case I need to have the same piece of code deal with items that are either instances of Django models or plain dictionaries, so I need to provide a uniform way of getting at the attributes.

Turns out it's pretty simple:

class DictObj(object):
    def __init__(self, d):
        self.d = d

    def __getattr__(self, m):
        return self.d.get(m, None)

d = DictObj(d)
d.name
# prints Joe
d.mood
# prints grumpy

"Beanstalkd / Python Basic Tutorial"

2008-10-12T10:00:00-07:00

(First install beanstalkd and pybeanstalk)

Beanstalkd is an in-memory queuing system. It supports named queues (called 'tubes'), priorities, and delayed delivery of messages.

Terminology: a message is called a job, and queues are called tubes.

Let's look at an example scenario. Say you want to create 2 tubes, one called "orders" and another called "emails", place orders into the first tube (or queue) and emails into the second, and have different processes handle orders and emails.

You can create queues by simply naming and putting messages into them. On the producer side:

from beanstalk import serverconn
c = serverconn.ServerConn('localhost', 99988)

# put a message (or job) into the default queue:
c.put('first message, into default tube')

# now start using a named tube:
c.use('orders')
c.put('second message, into orders tube')

Now on the consumer:

from beanstalk import serverconn
c = serverconn.ServerConn('localhost', 99988)

# by default your connection will be listening on the 'default' tube.
# switch it to use the 'orders' tube.
# This should return the 'orders' message and ignore the 'default' message:
c.watchlist = ['orders']
j = c.reserve()
print j
# {'data': 'second message, into orders tube', 'jid': 39, 'bytes': 32, 'state': 'ok'}

You can similarly setup another consumer to listen on only the 'emails' tube, or both, or any other scenario you want.

Beanstalkd also supports priorities, with 0 being highest priority and higher numbers meaning lower priority. You define message priority with:

c.put('low priority message', pri=999 )
c.put('high priority message', pri=0 )

j = c.reserve()
print j
# {'data': 'high priority message', 'jid': 41, 'bytes': 21, 'state': 'ok'}
# the high priority message was delivered before the low priority message, even
# though the low priority message was first into the queue

The beanstalkd consumption model is to "reserve" a message (or job), process it, and then tell beanstalkd you've successfully dealt with the message so it can be thrown away. When you first get the job via c.reserve() you haven't actually fully consumed it; you've just reserved it for processing.

What does this mean? Imagine a scenario where you reserve a message but your process dies before you have a chance to fully process it. Beanstalkd holds your message in reserve for a period of time, but since it hasn't heard from you confirming you've successfully dealt with the message, it eventually removes the reservation and makes the message available once again for the next consumer to grab. This is a basic handshake between the consumer and the server to allow for some level of resiliency.

So once you're finished dealing with the message you've reserved, be sure to "delete" it, letting the beanstalkd server know it can throw that message away:

j = c.reserve()
# do some processing with j
c.delete(j['jid'])

So there you have the basics. Let me know if you're interested and I can cover a few more topics.

"Setting Up Beanstalkd on Ubuntu for Python"

2008-10-12T10:00:00-07:00

beanstalkd is a promising in-memory queuing system in the mold of memcached (minimal configuration, just works) with client libraries in a variety of languages. The following worked for me for installing it on Ubuntu 8.04:

mkdir ~/packages

# pre-requisite: libevent.
cd ~/packages
wget http://monkey.org/~provos/libevent-1.4.8-stable.tar.gz
tar zxvf http://monkey.org/~provos/libevent-1.4.8-stable.tar.gz
cd libevent-1.4.8-stable
./configure
make
sudo make install

# add /usr/local/lib to your load library path so beanstalkd can find libevent
vi ~/.bashrc   (add the following somewhere near the end):
export LD_LIBRARY_PATH=$LD_LIBRYARY_PATH:/usr/local/lib

(exit vi)
source ~/.bashrc

# need git in order to get latest code for beanstalkd
cd ~/packages
sudo apt-get install git-core

# grab beanstalkd
git clone http://xph.us/src/beanstalkd.git
cd beanstalkd
make

# now you should be able to start the beanstalkd daemon
./beanstalkd -d -p 99988

# get the python beanstalkd client
cd ~/packages
svn checkout http://pybeanstalk.googlecode.com/svn/trunk/ pybeanstalk-read-only

cd pybeanstalk-read-only
sudo python setup.py install

# get pyyaml, a pre-requisite for the python beanstalkd client
cd ~/packages
wget http://pyyaml.org/download/pyyaml/PyYAML-3.06.tar.gz
tar zxvf PyYAML-3.06.tar.gz
cd PyYAML-3.06
sudo python setup.py install

# open two different shells (or use screen) type the following in the two different shells:
cd ~/packages/pybeanstalk-read-only/examples
python simple_clients.py producer localhost 99988
python simple_clients.py consumer localhost 99988

"UCSD Hack Week: I’ll Be There"

2008-10-12T10:00:00-07:00

I'm participating in the UCSD Yahoo Hack U week this coming week (Oct 13th-17th) with a talk on Wed and hopefully sticking around through most of the actual hack day (Thurs/Fri). If you're going to be on campus let me know by emailing me ( darugar at gmail ) or leaving a comment below.

"Back Up No More?"

2008-10-10T10:00:00-07:00

My laptop suddenly stopped working, throwing me into panic - what if it's a problem with the motherboard, the same problem that killed my last laptop?

It turned out to be the power adapter, fairly easily remedied, but it got me thinking about backups.

I realized I actually don't have a lot to backup. All of my important work stuff is already under version control, available from another server. Losing my pictures would suck, but the majority of the ones I like I've already put on Flickr.

My email is already on gmail and Yahoo mail, as are my contacts and calendar. My personal documents are mostly on Google docs, and the others are available from various servers here and there.

Just about everything I do is already in the cloud. Which makes it very nice - practically speaking, if I had to rebuild my working environment the majority of the effort would go into setting up the various development tools instead of recovering things from backup.

Quite nice, and quite different from 5 years ago.

"Thankful For The Stupidity of Virus Authors"

2008-10-10T10:00:00-07:00

For about a week now the network at home has been slow. Just as I'd get frustrated enough to do something about it it'd get fast again, so I procrastinated.

Meanwhile I was cursing Time Warner, Linksys, Apple, and just about everybody else whose hardware I own. The Linksys wireless router is 5 years old, but come on, it should still work!

This morning my wife complained of web access on the desktop being slow, so I sat down to take a look. Sure enough, very slow. It was odd that the computer itself wasn't as responsive as it used to be, but then it is a windows box, so curse Microsoft.

As I was dialing Time Warner to schedule someone to come out and take a look I started to look for a bandwidth tester so I could prove how crappy their service was. Oddly the search results didn't actually go to a bandwidth meter, but rather to an intermediary page with lots of links to bandwidth meters.

And that's when I realized: I gots me a virus. Fantastic. It's sitting there eating my bandwidth, probably spamming the world.

Point is, I was ready to blame everyone in the world instead of thinking it's a virus. If the virus authors hadn't put in the search redirection they could've probably lived on for another month sending their merry spam.

I'm certain a clever virus could infect a very large percentage of the world's computers without anyone noticing. Probably already has.

"Stackoverflow: Surprisingly Good Source of Technical Answers"

2008-10-09T10:00:00-07:00

Stackoverflow is a newish service for developers - ask a question, get answers, vote on answers, build reputation. Sort of like Yahoo! Answers but for developers.

I'm surprised at how good the service is so far. I asked a question on IRC and the same question on Stackoverflow. Within 2 minutes I got an incorrect answer on IRC and the two correct answers on Stackoverflow. The voting and reputation seems to really work and there's no IRC trolls / egos to deal with.

I'm hoping the site will maintain its usefulness as it grows. Well worth checking out. I'm here.

"The Big Picture Adopts ReaderScroll Navigation"

2008-10-07T10:00:00-07:00

This is an excellent development - The Big Picture from boston.com, a beautiful photo blog and one of the big reasons for putting ReaderScroll together, has implemented j/k navigation, inspired by ReaderScroll. Nicely done - now you can navigate the pictures via the j/k keys.

One down, many more to go. First step towards j/k navigation everywhere.

Btw, several people have pointed out vi used j/k navigation before Google Reader. True enough. j/k was also used in very old keyboard based games. However, the use of j/k to center web content was first brought to my attention in Google Reader, so I'm sticking with "Google Reader Style" instead of "vi style".

"Depressed"

2008-10-02T10:00:00-07:00

Watching the Palin/Biden debate and the subsequent babbling by the TV talking heads has me depressed. Nothing of substance was said during the entire debate. What we witnessed was an audition for a supporting role in a B movie, not a debate between potential 2nd in commands of the most powerful country in the world.

Apparently these folks haven't noticed, but we're in the midst of the most serious economic downturn since the great depression. We're neck deep in two active wars. There are serious problems to solve. Are sound bites, repeated claims of "maverick", and milf references really the best we have to offer?

What is depressing is not that these folks talk in pre-cooked, substance-free parrotings, but that we really and truly are stupid enough to not only tolerate, but to encourage this.

Please, give me something to vote on beyond how folksy, how old, how female, how dark, and how maverick these people are. I'm not trying to pick the person who looks the most like the president, but the one who will act the most president.

Bah. Humbug.

"Nookiller"

2008-10-02T10:00:00-07:00

Is it me or does Palin struggle to say "nookiller" instead of nuclear? It seems like she knows the correct pronunciation but goes out of her way to say it incorrectly. Some sort of Bushism or republicanism maybe?

Actually it'd be interesting to draw party lines along pronunciation. I say potato, you say potAto.

"Happy: Hadoop with Python (Jython)"

2008-09-24T10:00:00-07:00

The Freebase folks have open sourced their Python (Jython) based Hadoop framework, calling it Happy. Looks interesting, will need to give it a whirl when I get a chance.

"Best Way To Automatically Convert Flickr Images to FaceBook Albums?"

2008-09-22T10:00:00-07:00

Here's a lazyweb question for you: what's the best way to convert my flickr photo stream into facebook albums on an automated, continual basis? I much prefer dealing with flickr than facebook for photos, but many of my lazy friends are on facebook, keep asking for photos, but are apparently unable to click over to flickr or subscribe to the RSS feed.

There should be an easy way to have my photos automagically show up in facebook, right? I tried the top 2 flickr apps in facebook but they don't seem to do this.

"Productivity Tip: Forget The Power Cord"

2008-09-17T10:00:00-07:00

How's this for a productivity tip: leave your power cord at home. I'm in this situation now, and I'll tell ya, it does bring a razor sharp focus to what you need to be doing with those precious few minutes you have of computer time.

(Of course if I was being really productive I'd be working instead of posting this)

"Disco: Erlang/Python Based Map-Reduce"

2008-09-15T10:00:00-07:00

Disco is a map-reduce framework written in Erlang and Python. Seems reasonable - I definitely prefer Python to Java for writing maps and reduces, and Erlang is rumored to be good at parallel stuff.

Interestingly no mention of an underlying distributed file system.

Via High Scalability.

"Live Maps Rocks"

2008-09-15T10:00:00-07:00

Live doesn't get a lot of love and with those new ads Microsoft needs some serious help getting the word out, so let me say something nice about them. I've been using Live Maps lately, mainly because of the "Bird's eye view", and it's quickly becoming my favourite map app. If you have't tried it give it a shot: http://maps.live.com/ . I'm guessing you'll like it.

"Already Using Chrome Regularly"

2008-09-09T10:00:00-07:00

I'm surprised to find I use Chrome pretty regularly already. It's mostly my interface to Gmail, Google Reader, and GCal, and sits quietly in the task bar until I need to check email. Lack of firebug and various other extensions prevent me from using it for everything, but it's part of the regular line-up of apps.

"Drizzle: MySQL Based Slim / Cloud-Oriented DB"

2008-09-05T10:00:00-07:00

Drizzle is interesting:

Drizzle: A High-Performance Microkernel DBMS for Scale-Out Applications
Drizzle is a community-driven project based on the popular MySQL DBMS that is focused on MySQL's original goals of ease-of-use, reliability and performance.

Headed up by Brian Aker, Director of Architecture at MySQL AB. Take a look at the MySQL Differences page and you'll mostly see features removed and cleaned up, which is great. Designed for high levels of concurrency, targeted to "cloud" applications. Monty and Brian's posts offer motivation for the project.

Something to keep an eye on.

"Python Generator Expressions"

2008-09-04T10:00:00-07:00

I wasn't aware of generator expressions:

wwwlog     = open("access-log")
bytecolumn = (line.rsplit(None, 1)[1] for line in wwwlog)
bytes      = (int(x) for x in bytecolumn if x != '-')
print "Total", sum(bytes)

Similar to list comprehensions, but evaluated lazily. Voidspace describes it:

None of the generators are consumed until the final call to sum. As it iterates a line at a time (not keeping the log file in memory) it can handle huge log files - and as a bonus it runs faster than a typical solution with loops!

Neat.

"Trying Google Chrome, So Far So Good"

2008-09-02T10:00:00-07:00

The Chrome download was surprisingly fast (as in, less than 5 seconds). FireFox options import during install didn't work, but it's been smooth sailing since then. It's snappy and so far not buggy. Aesthetics of rendering web pages is good, seems better than FireFox. Quite fast too.

I remember thinking the Konqueror guys were crazy for writing yet another HTML renedering engine. That turned into WebKit, and now it's in Safari, the iPhone, and Chrome. Amazing.

"Google Chrome: Based on WebKit instead of Mozilla"

2008-09-01T10:00:00-07:00

Interesting that Google's Chrome is based on WebKit instead of the Mozilla rendering engine…

"Javascript Inheritance"

2008-09-01T10:00:00-07:00

Javascript inheritance is another one of those topics that always escapes my mind, so I'm recording it here for future reference.

I tried to get into Crockford's Prototypal Inheritance method but I like the plain-old prototype inheritance without the object.create better. It's entirely possible that I'm not quite getting Crockford's method and its advantages; feel free to enlighten me by leaving a comment.

Anyway, here's what I'm going with: We'll have a base class (actually an object) English that says "hello" and "goodbye". We'll have a Spanglish class (actually an object) that inherits from English and says "ola" instead of "hello" (over-rides the hello method), but keeps saying "bye" (inherits the bye method from English).

The Spanglish.prototype = new English; line does the actual inheritance by setting Spanglish's prototype to be an instance of English.

function English() {
    this.hello = function(){ return "hello"; }
    this.bye   = function(){ return "bye"; }
}

function Spanglish() {
    this.hello = function() { return "ola"; }
}
Spanglish.prototype = new English();

e = new English();
s = new Spanglish();

e.hello();   // prints "hello"
e.bye();     // prints "bye"
s.hello();   // prints "ola"
s.bye();     // prints "bye", inherited from English

"Being Smarter About Car Repair"

2008-08-27T10:00:00-07:00

What I know about the innards automobiles I learned from a book on Wankel engines when I was 8. So when my "check engine" light turned on I knew I'd be in for a severe siphoning of money out of my pocket.

Buoyed by my strong sense of miserliness, however, I decided to take a stand and brave the world mechanical. And I'm proud to say I managed to change a broken O2 sensor on my Honda Accord without having to visit a mechanic (or, to be more accurate, watched as my dad changed it, but hey, I was there).

Here's the key trick: all cars built since 1996 have a standard system called OBD-II (On-Board Diagnostics) that monitors much of your car and its health and reports standard codes for any mis-behaving components. There are code readers available from your favorite retailer (here's one from Amazon for $62) or to borrow from certain auto repair shops.

Connecting the reader to car is generally easy (it was simply a matter of attaching it to the connector under the steering wheel in the Honda) and you can find explanations for the codes online.

Armed with information about what exactly is wrong with your car you can have a much more … productive conversation with your mechanic.

Or, you can fix it yourself, as we did. Changing the O2 sensor turned out to be surprisingly simple, and not only did I save myself about $300, I also feel extremely proud of myself. I think this is my crowning achievement of 2008. Handy, I am.

Image courtesy of MadMan the Mighty.

"Google Reader Style Hotkeys For Image Viewing"

2008-08-25T10:00:00-07:00

I frequently run into pages with loads of images (here's one via Kedrosky, here's another, and here's yet another). Viewing these images involves a dance of scrolling or hitting space, then scrolling again to get the image lined up in the viewable part of the browser, and repeating over and over again.

The better user experience would be to have Google Reader style hotkeys that bring each picture into view, lining it up properly for easy viewing. You'd be able to hit "j" after viewing each picture to see the next. No more messing with the scrollbars.

I'm guessing a greasemonkey script to enable this with any page would be pretty straight forward to do. Anybody know if it already exists? If not, someone should do it.

"Three"

2008-08-25T10:00:00-07:00

Three.

The number of programmers who will write most of the code in a system developed by a team of 24 engineers, two project managers, three group leaders, a quality lead and an office manager.

From Russ Olsen.

"Startup VP Technology / CTO Opportunites"

2008-08-23T10:00:00-07:00

Several friends have been at work on several startups and are looking for technology leaders to round out their teams. One opportunity is in the social network / eduction space and is looking for the VP Technology / CTO as well as good web/php guys. Another is more hardware related. Both are very viable early stage companies in San Diego. If you have interest or know of people who might let me know - darugar at gmail .

"Terrorists: There Is No Profile"

2008-08-22T10:00:00-07:00

MI5 report on terrorist profiling, via Schneier. The short version: it is very difficult to come up with a valid profile.

"TraceMonkey: It’s A Big Deal"

2008-08-22T10:00:00-07:00

Mozilla announces TraceMonkey, a just-in-time compiler for Javascript. If you've watched Steve Yegge's talk on Dynamic Languages (transcript) you've already had a taste of what the future could look like for dynamic languages - namely, performance on par with today's low level languages.

Javascript started as an ugly language but has been steadily shedding its bad parts and adopting a beautiful functional style. With the performance piece figured out and a tremendously large number of installations and runtimes (just about every browser in existence has a Javascript engine), it could become the most important programming language of the near future.

"Awesome"

2008-08-21T10:00:00-07:00

Parand.

"A Beautiful Python Twitter API"

2008-08-18T10:00:00-07:00

Mike Verdon's Python Twitter Tools is less popular and findable than the Dewitt Clinton's python-twitter (I only found the former from an email on the latter's mailing list), but it's a beautiful library. 125 lines, most of which are comments. It implements the full API by implementing a single call class that handles everything (and that class is only a few lines). A good library to use if you're trying to access Twitter via Python, and a good source for learning and inspiration if you're writing a library / wrapper.

"Sickening Olympic Coverage"

2008-08-18T10:00:00-07:00

Watching the women's uneven bars. The announcers have spent the vast majority of the time whining and moaning Nastia's the 2nd place finish.

Now I recall earlier gymnastics events where these same announcers said starting first is a huge disadvantage and effectively penalizes the American competitor (pretty sure it was Shawn Johnson's event). Hmm. The Chinese winner of this event went first. I didn't hear any whining about how she was at a disadvantage.

We're also hearing about the incompetency of the judges. Apparently the Australian judge favors the Chinese. Makes sense that Australia would pick China over US. Eh. I'm sure it does somehow. Nevermind that his score was actually thrown out (top and bottom scores are thrown out).

The tie-break procedure is also apparently anti-U.S. I've heard Al-Qaeda was behind instituting it.

Watching the track events earlier I was shocked to see there were 8 athletes running in each heat. The announcers announced 2, and the camera was very clear that there were only 2 athletes running. Turns out there are other countries competing besides America. Which really pisses me off - don't they know it's ALL ABOUT AMERICA?

I'm sick of NBC. These guys are making America look like a bunch whining, sore losers, which is anything but the truth. The athletes are doing a fantastic job and the venue is beautiful. If only these NBC idiots would embrace the spirit of the Olympics and realize it's not all America versus the rest of the world. It's a celebration of the athletic spirit of the world.

There we go. And they finish with how America's dominance of baseball and softball is so complete the Olympic committee had to drop them from the Olympics.

Think back to high school, to the rich kid who was a good athlete and had everything else going for him, but was still an asshole? There you go. Apparently that's who we are. Despite everything we have we still can't find it in ourselves to be gracious.

Damned Australia.

"Sweet Sleep, Part Deux"

2008-08-17T10:00:00-07:00

Wife is in Florida with the youngest kid, and the other two spent most of the weekend at my parents', giving me a chance to engage in plenty of bad habits. Mostly work and sleep.

It's an amazing feeling to wake up after more than 8 hours of sleep. Really quite fantastic. It's obvious with the kids - if they don't get enough sleep they're cranky and lethargic. The same is at work with adults, we're just better able to mask it.

Anyway: sleep. I highly recommend it.

"Trying Mercurial"

2008-08-17T10:00:00-07:00

All the cool kids are into Git these days and I've been reading plenty of articles about how good it is and how to use it. The problem is, I don't really have a problem with Subversion. I know I should, because all the cool kids do, but I just don't run into a lot of issues with it. In the absence of a problem to solve it would simply be peer-pressure to give Git a shot.

So in an attempt to remain ever-so-independent I'm going to try a distributed system, but not Git. I'm going with Mercurial.

Actually mainly it's because Mercurial seems significantly simpler, and I'm a simple guy. It's also written in Python, which gives me a warm and fuzzy. And I'm finally motivated to try it because I'm going to try a code path which may not work out, and I understand these distributed systems deal with that well.

Hmm. The main thing I'd want from a source code control system would be a bit of packaging and deployment intelligence built in. Maybe something to minify and join my javascript files and mend the files that reference them. I'm extremely pleased not to have a "make" step anywhere in my process, but I do miss some of the capabilities.

If I'm making a mistake and I should go with Git, or perhaps CVS, do let me know.

"Amazing Video Enhancement / Transformation Technique"

2008-08-16T10:00:00-07:00

This is just incredible. The effects are amazing, but I'm guessing the models and tools behind what they do are going to fundamentally change the way we use video. Apparently these guys are divining a pretty good depth and color projection map from videos and photographs. With that you've added entirely new dimensions to the readily available information conveyed by video, opening the door to a huge range of effects and enhancements. Looking forward to seeing more of this.

"Taste Vectors: OpenTaste"

2008-07-18T10:00:00-07:00

I couple of years ago I wrote about Taste Vectors, did a bit of hacking, and not much else.

The Strands folks have launched a similar concept in OpenTaste. Worth a look, will write more if I get a chance.

Via Duke Listens!

Parand

The Bear Case for Frontier AI Labs

The AI Productivity Lie

Don’t Be A Tourist In Your Own Code Base

Or: Vibecoding versus LAD

Write The Spec, Let the AI Design?

Is It Just Me?

Terrible At Architecture

Big Picture Blindness

The Rabbit Hole

Plain Stupidity

What Works? LAD

Fun and Profit

Why Do Large Language Models Hallucinate?

What Does Hallucination Mean Here?

Why Lie, ChatGPT?

There's No Database

What's the Fix?

Further Reading

A Non-Technical Explanation of ChatGPT: Deep Learning

Creating CatGPT

Learning, Deeply

Eureka: We can Train

Deep Learning For Hard Problems

CatGPT Works. Sort Of

Attention Please

Transformers and Attention

Cat Communication

Human-in-the-Loop and Fine Tuning

CatGPT

Further Reading

A Non-Technical Explanation of ChatGPT

Fill In The Blank

Aliens and CatGPT

Markov to the Rescue

Large Language Models

To be Continued...

Further Reading

The Parable of the Physics Professor

A Completely Non-Technical Explanation of AI and Deep Learning

Overview

Day One

Day Two

Day Ten

Neural Networks

Day Forty

Deep Learning

Day Fifty

Convolutional Neural Networks

Further Reading

A Practical Introduction to Polars for Python Data Processing

The Setup

The Data

Polars

Most expensive rides

Zone Names

Data Exploration and Cleansing

Basic Cleaning

Finding bad pairs

Links

Blog Technical Setup

Overview

Everything Sad is Untrue: Book Review

Parable of the Sower: Book Review

Banshees of Inisherin: Review

"Python itertools.groupby : a clearer example"

"Undo last git commit / merge"

"Efficiently finding large (size and dimension) images"

"Hands On With the BlackBerry Playbook: Actually Quite Nice"

"SFTP: Specifying Port, SSH key"

"Student Loans Are Killing Entrepreneurship"

"How To Dynamically Import Python Modules"

"How To Scan Multiple Pages Into a Single PDF on OS X"

"Titanium Studio: How To View Android Log Messages"

"Easy way to view cookies in Chrome"

"Uniforms"

"Future of Work and New Dial Tones"

"Generating static html from markdown in python"

"Retention Bonuses Don’t Work"

"mo.com Interviews Me"