The Solution

WebDev’s crisis is a common sight for us. This is what happens when technical issues can’t be solved. Technical issues are one of the few problems that can bring projects to a grinding halt. The problem with technical issues is that there is no guarantee that you will be able to solve them. Just because you throw more bodies at the problem doesn’t make it go away faster. It is the biggest reason why companies fail: they could not execute. This is the ugly side of business that people don’t like to talk about but happens all the time.

Technical issues are like a virus that slowly takes over a company until it kills it. It starts first with the engineering team where failure kills morale and makes their performance worse. Engineering missing targets ruins plans for other parts of the company like management, operations, and marketing who now have new issues to deal with on top of their existing ones. Customer service then has to spend all its time dealing with angry customers. Soon the damage to relationships is irreparable. What starts as a simple engineering task could ultimately create the downfall of a company.

With the situation spiraling out of control, we had to stop the bleeding and save WebDev and their client.

Speed

When we analyzed their work, we saw fundamental flaws in their approach. The flaws made it impossible to create anything usable. These issues were apparent to us on the first day. They were fixed in one day. Months of headaches and pain was turned around in a single day.

Data Model

They tried to create a flexible data model to speed up development, but their simple model killed the performance. They were using meta-based one-to-many storage locked in by the framework. This was really flexible because it allowed infinite, dynamic expansion of fields on entities so they could change their mind whenever they needed to, making development easier.

The problem is that the entities they were working with had up to 40 fields. That meant 40 joins to get a complete entity. A table of 10,000 entities took up 400,000 rows. Entities had relations to multiple other entities which each had up to 20+ other fields making getting complete object set a scary thought. Indexing and sorting were incredibly slow.

What is easy to build is not always the best to build. We knew this model had to be ditched. We replaced this data model with a leaner flat model. It was indexed properly for fast performance. Transitioning to the new model completely would take too much time, so we created a bridge between the two during migration. What took 40 hours before took less than 5 minutes now, a speedup over of 480x.

Overhead

They were using default framework commands. Frameworks often provider helper functions to do common things like insert and update objects. The issue is that these functions are rarely ever scalable. These functions often call other functions and perform extra queries to check for certain things like data validity. This creates massive overhead. They work for very small use cases with a couple hundred items but are disastrous on larger systems. They had almost 200k and would need to scale to millions.

People use framework commands when they do not understand the underlying data model. Inserting objects in multiple linked tables the way the framework expects it is tricky and dangerous. Doing it wrong can easily destroy a site, so people just avoid it. They’ve only used high-level abstractions instead of raw code. The trade-off is severe performance issues.

We were able to monitor what their framework was doing when it inserted objects and replace the framework commands with our own leaner commands that bypassed the unnecessary checks. An insertion with commands was taking 8 seconds. With our leaner, we reduced that to less than a hundred of a second, a speedup of over 800x on insertions.

Geolocation

The geolocation issue stemmed from a third-party plugin they were using but did not know how to set up. They did not understand how to get the plugin to work on the client’s servers. In their internal testing at their company offices, they were using pre-built pseudo-servers like XAMPP that already had things set up for you. So on a real server, they had no idea how to do it and could not instruct a server team on what to do.

We do not use third-party code unless we need to for a good reason. The reason why is because they carry liabilities that you will have to deal with at some point whether you realize it or not. This plugin was sending requests to external servers for geo calculations when we could emulate the data locally and get a major performance boost. So we ditched the plugin and wrote our own geolocation module.

The crucial step they were missing was setting it up on a live server. They brought their entire company of almost 40 engineers to the project, but no one could figure it out. The answer lied in the fact they didn’t understand how to work with a user’s real IP behind a proxy. This took us a day to figure out. Once again, months of hassle solved in a day.

Crisis Averted

With major technical features quickly working, their client completely changed their tone. They were so happy to see progress and a real future for their company. With things working, there was less pressure to recoup related losses from WebDev. A major lawsuit no longer seemed inevitable, saving WebDev from potential corporate liquidation. Peace, it seemed, was finally possible.