Why are developer estimates almost always short?

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/why_are_developer_estimates_almost_always_short.htm]

We constantly need to make estimates in software engineering. But, the ironic thing is not that our estimates are off, but rather that they're almost always short. There's a one-way bias.

I think this is because:

• It always looks simple in our heads. Devs are overly optimistic and plan for best case scenario, and hence get caught off-guard. I mean, who would ever guess that it takes 5 hours to fix a single line of code?
• Scope creep - The business just keeps piling on more requests (and it's hard to say "no" to the business).
• Strong bias from managers - Most managers want the lower estimate because, regardless of whether the estimate is actually accurate, it's easier to sell to their own boss or clients. For example, in consulting it's easier to give a lower estimate to "get your foot in the door", and then gradually try to pile on more services.
• Devs try to impress - I can't help but wonder if many devs try to impress their managers by providing a lower estimate "Sure Mr. Burns, I can get that 5 day task done tomorrow (because I'm a rock star developer... please appreciate me)"
• It's almost expected - At least in my experience, it almost seems like (unfortunately) the industry has resigned itself to low-ball developer estimates. So the managers always just double (if the estimate is from a senior dev) or triple (if the estimate is from a junior dev).

The best solution I can think of: read and study Steve McConnell's phenomenal book on Software Estimation: Demystifying the Black Art.

Also see: How not to estimate, Playing estimates off each other.

Civil Engineering and Writing New Code

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/civil_engineering_and_writing_new_code.htm]

I have a special appreciation for civil engineering - I just find the bridges, highways, dams, and sky-scrapers beautiful in a way that an engineer can appreciate.

In America, before all the infrastructure was built, one might say that there was a "golden age" of civil engineering. With literally millions of square miles of country, there were seemingly endless opportunities to build new structures. And at the time that these structures needing building (before computers, flight, or bio-engineering), civil engineering was arguably one of the advanced fields of its day. You put these circumstances together: lots of new projects that require advanced technology - and you've got an engineer's dream.

Of course, over two hundred years, enough roads and bridges and buildings were built to fulfill much of the country's need. There were a couple spurts in between - building the highways, the major airports, an occasional monument, and more recently the cell phone transmission towers. However, in general, the civil engineering mindset went from "new creation" to "infrastructure maintenance".

At least from my perspective, the same life-cycle appears to be happening with software engineering. Even back in the 80's, just finding a machine, learning the syntax and writing a "program" was a new thrill. However, especially with the internet (just google the syntax and copy code-snippets), better hardware (you don't need an algorithm-genius to make an application perform), mass availability of machines, outsourcing (huge pool of developers), standard platforms and components that encourage reusability instead of re-inventing the wheel, and simply enough years passing - almost every established company has some sort of core IT infrastructure in place. Back in the late 90's, major companies had huge booms to implement their ERP, email, and websites (made lots of consultants very rich) - but now those expensive systems are in place. Sure, there's always work to do, like integrating components, migrating code, consolidating applications, extending functionality of existing apps, and maintaining existing code. There's still new development, but it seems much scarcer than 10 years ago. The cowboy days of just being thrilled to write a program seem to have passed.

Similar to how civil engineering has filled the country's physical infrastructure, software engineering has filled much of the country's IT infrastructure - and therefore in both cases much of the work currently being done is maintenance. America doesn't make many Hoover Dams or Golden Gate Bridges anymore - but there's always annual road re-surfacing. Same concept for developers. (This means that there's tons of legacy code out there, which is why every developer should read the phenomenal book Working Effectively with Legacy Code.)

Some developers view this in a pessimistic light, as if "the good old days" have passed us by. However, I'm an optimist, and there's much reason to believe that there are still many innovative and new software development efforts ahead.

• There are continually newer technologies - This provides a business incentive to rebuild older systems. Web systems replaced many fat clients. But now web 2.0 is replacing many existing web systems, and mobile apps may replace those, and there will be something after that (what if voice recognition takes off and makes most UI obsolete?).
• Much room for innovation - The nature of IT (with the low barrier to entry, the ability to cheaply experiment, and building projects out of "thought" stuff) allows for massive innovation, unlike any other field I can think of. Innovation means hopes for a better, more profitable system, and therefore business sponsors to fund new development.
• Software applications have a short lifespan. - Most software projects are replaced within 5-10 years, so there is continually new work. (A good bridge or building could last for a hundred).
• Programming is a foundational skill that can assist you with everything else - Because almost every desk job in any industry uses MS Office apps (or the equivalent of), databases, and web sites, the ability to write even simple programs can assist with those jobs. For example, writing an Excel macro or finding results in a SQL query may let you get a task done much quicker.

So while on one-hand there's definitely more maintenance work for all the legacy systems, as the field of software engineering matures, I think there's still a lot to be optimistic about for new development opportunities.

Microsoft Office Lullaby

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/microsoft_office_lullaby.htm]

Somehow, after having three kids, I started singing this kind of lullaby (sung to the classic theme "Hush little baby"):

Hush little baby, don't say a word, Daddy's going to teach you Microsoft Word.

And if that MS Word does not spell, then Daddy's going to teach you Microsoft Excel

And if that Excel doesn't save the day, then Daddy's going to teach you VBA

And if VBA doesn't hit the mark, then Daddy's going to teach you how to program C#.

Chicago Code Camp this Saturday

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/chicago_code_camp_this_saturday.htm]

This Saturday is the 2010 Chicago Code Camp (Last year's rocked!) For those who have a passion for coding, this is the event for you!

 

Technology is the engine, but the Business is the steering wheel

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/technology_is_the_engine_but_the_business_is_the_steering_w.htm]

I like analogies. Initially, I was thinking of polar coordinates, where technology is the magnitude, but the Business is the direction - however that sounded too techy.

The idea is that you can have the most powerful technology in the world, but if it's not focused in a profitable direction, your business (which pays for the technology) will never get to the right place.

You need both tech and business, else you plateau very quickly. Without business understanding, the technologists can build amazing things - that no one wants. That's fine for hobbyists and CS college students, but not sustainable for companies.

Without technical understanding, the business people can dream of profitable ideas - that no one can build.

Unit testing random methods

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/unit_testing_random_methods.htm]

You can unit test random methods by running the method in a loop and checking for statistical results.

For example, say you have a method to return a random integer between 1 and 10 (this could just as easily be to return any type between any range). You could run the test 100,000 times and confirm that the statistical distribution makes sense. With a sufficient sample size, there should be at least one of each value. The mathematically advanced could apply better statistics, like checking the proper distribution.

Here's a simple sample. It runs the random method in a loop, and checks just that each value was returned at least once.

public class MathHelper
{
    private static Random _r = new Random();

    public static int GetRandomInt()
    {
        //return Random int between 1 and 10
        //recall that upper bound is exclusive, so we use 11
        return _r.Next(1, 11);
    }
}

[TestMethod]
public void Random_1()
{
    int iMinRandomValue = 1;
    int iMaxRandomValue = 10;

    //Initialize results array
    //ignore 0 value
    int[] aint = new int[11];
    for (int i = iMinRandomValue; i <= iMaxRandomValue; i++)
    {
        aint[i] = 0;
    }

    //Run method many times, record result
    //Every time a number is returned, increment its counter
    const int iMaxLoops = 1000;
    for (int i = 0; i < iMaxLoops; i++)
    {
        int n = MathHelper.GetRandomInt();
        aint[n] = aint[n] + 1;
    }

    //assert that each value, 1-10, was returned
    for (int i = iMinRandomValue; i <= iMaxRandomValue; i++)
    {
        Assert.IsTrue(aint[i] > 0,
            string.Format("Value {0} never returned.", i));
    }

}

Of course, this assumes you're explicitly trying to test the random method - you could mock it out if the method is just a dependency and you're trying to test something else.

LCNUG - Visual Programming Language with Robotics Studio

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/lcnug__visual_programming_language_with_robotics_studio.htm]

Last night, Lance Larson, president of the Madison .NET User Group, presented at LCNUG on Visual Programming Language (VPL) with Robotics Studio. It was an active presentation - he even had robots moving around the room!

Two things that really got my attention:

• VPL applies to more than just robots. Many businesses continually hit the problem "how can I have a non-technical person still get technical things done?" For example, they'd like a business analyst program a workflow or rules engine without needing to actually code. Many workflow-related products provide some kind of drag & drop interface (like making a flowchart in Visio) to effectively write a program, but such an interface is difficult to build. It gets especially complicated when you have variables, conditions, looping, etc... Microsoft's VPL is powerful, and I wonder if it will be reused for their workflow products too.
• Being surrounded by software, it's refreshing to see the hardware part of engineering - like physical robots that follow programming instructions.

Thanks Lance!