I was thrilled to participate in the 2011 Chicago Code Camp. We had about 300 people show up on Saturday to CLC, with 35 speakers, many volunteers, and a full day of learning and networking. Thanks to a dozen sponsors, the event was free to the public. This is the third year we've had a CCC, and it keeps getting better.
It really is a privilege to be part of such a community.
Sunday, May 15, 2011
Monday, February 28, 2011
4 Types of automated tests - unit, integration, UI, and performance
A software engineer could spend their life continually improving test automation - it's a big field. While the sky is the limit, there are at least 4 types of automated tests: unit, integration, UI, and performance.
Some key ideas:
Some key ideas:
- These build off of each other -
- Unit --> Integration: Don't bother with complex integration tests if you can't even have a simple unit test.
- Integration --> UI: It's going to be near impossible to do a UI test (which usually has poor APIs) if you can't at least integrate the backend (with at least has APIs - like web services, SQL, or C# calls).
- UI --> Performance: If you can't at least functionally run the code from end-to-end, then you can't expect reliable performance measures on it. Yes, there are always exceptions, and semantics (one may consider "UI" to be fronted integration, or one may test performance on just the backend APIs and bypass the UI). But in general, these 4 tests are a very natural path to follow.
- The higher you go, the more expensive: Unit tests (low-level) are cheapest, performance tests (high-level) are most expensive. So it's bad business to pay for an integration test to do the work of a unit test. It's like using an armani suit as a dish rag.
- These 4 types of tests should be separated. You can call any code from a unit test (depending on security, you could even call APIs to shutdown the server), so you could mix all your tests into one test harness. But don't do this - it will burn you. For example, unit tests are generally fast (they're all in-memory), whereas UI and integration are much slower (databases, web services, IIS hosts, etc...) So you don't want them coupled together because the slow integration tests will bog down your fast unit tests, and then developers never run unit tests before check-in because "it takes too long".
- Unit testing is but one tool. There is different types of code (algorithms, data containers, plumbing, installation scripts, UI, persistence plumbing, etc...). This requires an arsenal of developer skills, of which unit testing is one tool. With respect to unit testing and code coverage, the goal isn't N% coverage of all code, but rather N% coverage of unit-testable code. (You can get better coverage tools, like NCover, which can provide coverage when running integration, UI, and even manual tests run by QA, but that's a different story).
Test Type | Good For | Bad For |
Unit |
|
|
Integration ("backend") | Ensuring high-level flows work, such as you can call a web service that loads or saves data to a database and writes something to a file. | Anything that can be handled with a unit test instead. For example, you likely wouldn't use an integration test to verify every input combination for a text field. |
UI ("frontend integration") | Very-high level, functional tests. | Anything that can be handled via backend integration or unit tests. |
Performance | Identifying performance problems that could be costly to the business. | Any functional testing |
Sunday, February 27, 2011
Chicago Code Camp 2011 is coming in May!
The Chicago Code Camp (CCC) returns on Saturday, May 14, 2011. This has been very successful in the past (2010, 2009).
I think the 10 reasons to attend the 2009 CCC still apply, including the great content, and it's free.
Check out the CCC site: http://www.chicagocodecamp.com/
I think the 10 reasons to attend the 2009 CCC still apply, including the great content, and it's free.
Check out the CCC site: http://www.chicagocodecamp.com/
Monday, January 17, 2011
Gaming Unit Test Metrics
Unit Testing is a popular buzzword - most developer jobs request it, teams want to say they have it, and most coding leaders actively endorse it. However, you won't be able to hire a team of devs who don't want to write unit tests, and then "force" them via measuring certain metrics. Developers can game metrics.
1. Metric: Code Coverage
The #1 unit testing metric is code coverage, and this is a good metric, but it's insufficient. For example, a single regular expression to validate an email could require many different tests - but a single test will give it 100% code coverage. Likewise, you could leverage a mocking framework to artificially get high code coverage by mocking out all the "real" code, and essentially having a bunch of "x = x".
2. Metric: Number of unit tests
Sure, everything being equal, 10 tests probably does more work than just 1 test, but everything is not equal. Developer style and the code-being-tested both vary. One dev may write a single test with 10 asserts, another dev may put each of those asserts in its own test. Also, you could have many tests that are useless because they're checking for the "wrong" thing or are essentially duplicates of each other.
3. Metric: % of tests passing
If you have a million LOC with merely 5 unit tests, having 100% tests passing because all 5 pass is meaningless.
4. Metric: Having X tests for every Y lines of code
A general rule of thumb is to have X unit tests (or X lines of unit testing code) for every Y lines of code. However, LOC does not indicate good code. Someone could write bloated unit test code ("copy and paste"), so this could be very misleading.
5. Metric: How long it takes for unit tests to run
"We have 5 hours of tests running, so it must be doing something!". Ignore for the moment that such long-running tests are no longer really "unit" tests, but rather integration tests. These could be taking a long time because they're redundant (loop through every of a 1 GB file), or extensively hitting external machines such that it's really testing network access to a database rather than business logic.
6. Metric: Having unit testing on the project plan
"We have unit testing as an explicit project task, so we must be getting good unit tests out of it!" Ignore for the moment that unit tests should be done hand-in-hand with development as opposed to a separate task - merely having tests as an explicit task doesn't mean it's actually going to be used for that.
7. Metric: Having high test counts on methods with high cyclometric complexity
This is useful, but it boils down to code coverage (see above) - i.e. find the method with high complexity, and then track that method's code coverage.
Conclusion
Obviously a combo of these metrics would drastically help track unit test progress. If you have at least N% code coverage, with X tests for every Y lines of code, and they all pass - it's better than nothing. But fundamentally the best way to get good tests is by having developers who intrinsically value unit testing, and would write the tests not because management is "forcing" them with metrics, but because unit tests are intrinsically valuable.
1. Metric: Code Coverage
The #1 unit testing metric is code coverage, and this is a good metric, but it's insufficient. For example, a single regular expression to validate an email could require many different tests - but a single test will give it 100% code coverage. Likewise, you could leverage a mocking framework to artificially get high code coverage by mocking out all the "real" code, and essentially having a bunch of "x = x".
2. Metric: Number of unit tests
Sure, everything being equal, 10 tests probably does more work than just 1 test, but everything is not equal. Developer style and the code-being-tested both vary. One dev may write a single test with 10 asserts, another dev may put each of those asserts in its own test. Also, you could have many tests that are useless because they're checking for the "wrong" thing or are essentially duplicates of each other.
3. Metric: % of tests passing
If you have a million LOC with merely 5 unit tests, having 100% tests passing because all 5 pass is meaningless.
4. Metric: Having X tests for every Y lines of code
A general rule of thumb is to have X unit tests (or X lines of unit testing code) for every Y lines of code. However, LOC does not indicate good code. Someone could write bloated unit test code ("copy and paste"), so this could be very misleading.
5. Metric: How long it takes for unit tests to run
"We have 5 hours of tests running, so it must be doing something!". Ignore for the moment that such long-running tests are no longer really "unit" tests, but rather integration tests. These could be taking a long time because they're redundant (loop through every of a 1 GB file), or extensively hitting external machines such that it's really testing network access to a database rather than business logic.
6. Metric: Having unit testing on the project plan
"We have unit testing as an explicit project task, so we must be getting good unit tests out of it!" Ignore for the moment that unit tests should be done hand-in-hand with development as opposed to a separate task - merely having tests as an explicit task doesn't mean it's actually going to be used for that.
7. Metric: Having high test counts on methods with high cyclometric complexity
This is useful, but it boils down to code coverage (see above) - i.e. find the method with high complexity, and then track that method's code coverage.
Conclusion
Obviously a combo of these metrics would drastically help track unit test progress. If you have at least N% code coverage, with X tests for every Y lines of code, and they all pass - it's better than nothing. But fundamentally the best way to get good tests is by having developers who intrinsically value unit testing, and would write the tests not because management is "forcing" them with metrics, but because unit tests are intrinsically valuable.
Saturday, January 15, 2011
Writing tip: Concise language
You are "paying" for each word, and if your idea is too expensive to communicate, it will never be heard - let alone implemented. Therefore it's imperative that developers have good writing skills if they want to become influential. One metric of good writing is that it is concise. Here is a brainstorm of tips.
Tip: Remove un-needed words and phrases
If the word adds zero value, then remove it.
//Remove "Anything about"Anything about schemas will drive architecture.
Schema will drive architecture.
//Remove phrase "my approach"
I’ll need to confirm my approach with Bob
I’ll need to confirm with Bob
//Restructure sentence to remove the word "it"It takes effort to ask for something.
Asking for something takes effort.
Tip: The word is implied, so you don't need it.
//Implied that you're concerned, which is why you're talking about itMy concern is that X is bad.
X is bad
//The word "something" is impliedIf there’s something more to it.
If there's more to it.
//The word "about" is implied by estimate, and the number being rounded
We estimate the list to have about 10,000 rows.
We estimate the list to have 10,000 rows.
//The phrase "Ed mentioned" implies "Discussing with Ed"Discussing with Ed yesterday, he mentioned that...
Ed mentioned yesterday that...
Tip: remove duplicate words
Sometimes people use duplicate words. Maybe because (1) they think it drives the point home, (2)
they feel like it has to be long (schools trained them to write "at least 500 words"), (3) they
can't think of a single word to describe both sub-words.
they feel like it has to be long (schools trained them to write "at least 500 words"), (3) they
can't think of a single word to describe both sub-words.
//Just pick one - either "harsh" or "ruthless"The production environment is harsh and ruthless
The production environment is ruthless
Tip: Replace a phase with a single word
Sometimes a big phrase can be replaced with a single word because that single word is the definition of the big phrase.
//Replace the entire phrase with one word that means the same thingThe application must be able to handle an increasing amount of demand.
The application must be scalable.
Tip: Condense two sentences into one
//Remove a descriptive sentence by applying it to the noun directly.
I'd like to set up a call. An hour should be enough time.
I'd like to set up a one-hour call.
Tip: Avoid un-needed commas
If you have a bunch of small phrases joined by multiple commas, consider restructuring it into a single sentence. Even if it doesn't make it more concise, it will probably make it easier to read.
//Remove comma
and for each line, trims it.
and trims each line.
//Merge two short sentences into 1:
We are creating a custom list. It will have 1000 rows.
We are creating a custom list containing 1000 rows.
//Restructure sentenceYesterday, Ed mentioned that...
Ed mentioned yesterday that...
//Restructure sentenceIf you approve of this design, please let me know.
Please let me know if you approve of this design.
Monday, January 10, 2011
Writing tip: Avoid abstract words
We all know that influential developers need good communication skills. I love technology, but if you can't communicate your technical ideas to your coworkers, then they won't be adopted.
You need to be concise, else people will just tune you out. One way to do that is to choose a more descriptive work that packs more meaning. Your word count remains about the same - it's absolutely not just rambling on - but the sentence caries more meaning.
Of course sometimes you deliberately want the more abstract word, but it's good to be aware of the more-descriptive alternatives so that you can make a conscious choice of what's best for the situation.
You need to be concise, else people will just tune you out. One way to do that is to choose a more descriptive work that packs more meaning. Your word count remains about the same - it's absolutely not just rambling on - but the sentence caries more meaning.
Of course sometimes you deliberately want the more abstract word, but it's good to be aware of the more-descriptive alternatives so that you can make a conscious choice of what's best for the situation.
Weak word | Comment | Consider Instead | Example |
Do, Does | Does what? | create, build, make, install | Bob does builds Bob installs builds |
Run | What does the "running" do? | scrubs, cleans, updates, processes, installs, configures | The service runs on the directory The service cleans the directory |
Change | How does it change? | insert, update, delete, improves | The tool changes the file The tool updates the file |
Processor | What does the process do? | copier, uploader | FileProcessor tool FileUploader tool |
It, Object, Thing | What kind of thing? | C# class, SQL script, COM+ object | It runs fast. The C# class runs fast |
Talk | What kind of talking? | discussed, reasoned, concluded, negotiated, argued, explained | Bob talked about structs Bob explained structs |
Issue | What kind of issue? | bug, feature request, task, problem | This is an issue This is an error |
Other | In what way? | previous, next, remaining, bigger, smaller | I need the other node I need the previous node |
Good | In what sense? | faster, more reliable, higher revenue | This feature is good This feature is faster |
People | What kind of person? | manager, employee, developer, analyst, users | People will like this feature Managers will like this feature |
Sent | How? | FTPd, emailed, faxed | I sent the files I emailed the files |
Wednesday, January 5, 2011
They should not complain if...
Every project and team always has issues. I'm a big fan of making things better. No one wants to hear complaining (not to be confused with constructive criticism or a call for action). Here's a brain dump of when people should not complain:
- They don't have a better alternative
- They can't specify what the actual problem is (saying "something seems wrong, but I don't know exactly what" isn't complaining)
- If their manager asks "what can I do", and they can't tell them
- They caused the problem
- They were warned, and did it anyway
Subscribe to:
Posts (Atom)