10 tips to integrate CodeSmith into your processes

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/10_tips_to_integrate_codesmith_into_your_processes.htm]

Say you've theoretically seen why code generation is so profitable, so you've downloaded a free trial of CodeSmith, and banged out a few templates. In other words, you've got code generation working on a single developer machine. That's great, but it's even better to have it adopted by the entire department. Here are some practical tips on how to integrate CodeSmith into your processes.

1. Aim for active regeneration - There are two kinds of generation, Active and Passive. Active means that the code is actively regenerated on a regular basis. Passive means it was generated just once, and then modified manually thereafter. The problem with Passive generation is that it lets developers create tons of code upfront, but then people get trigger-happy and use the generator to produce even more code, and you're stuck now maintaining it all. It's like a trip with no return ticket. It also misses out on many of the other benefits of codeGen - like mass updating code with some new change.
2. Always have a batch script - Yes, people can integrate into VS, or use the CodeSmith IDE. But to enforce uniformity, ensure that the right properties are passed in, and hook into your Continuous Integration (CI) build, you'll need a batch.
3. Run the codeGen from your CI build - This enforces active regeneration.
4. Consider not checking generated scripts into source control - This prevents synchronization errors between local developers and the build server. Yes, all code should ultimately be checked into source control, which is why we still check the templates themselves in, from which you can deterministically recreate all the target code. Your automated checkout script, which gets the latest from source control, can then run the templates and recreate the target code. NOTE - this only works if you're not using merge regions (which mix generated and custom code in the same file). If you use merge regions, then you need to check in the generated files.
5. Avoid merge reasons where possible - CodeSmith has this powerful feature called "merge regions" which lets you mix both generated and custom code. Sometimes you need this, but if you have a choice - always opt to put generated code in its own, dedicated file. This prevents synchronization issues, is less likely to break, is easier to handle overwriting files in active generation, and is easier for most developers to understand and maintain.
6. Ensure that code generation can be run on every developer's machine - Because you'll want to actively re-generate the code, you'll need each developer to be able to run those generation batch scripts locally. That means each developer will need a license for CodeSmith. This is absolutely not the place to be stingy. If developers cannot simply make a change and have the code re-generated, they will revolt against using code generation.
7. Clearly identify the generated files - Make sure an average developer can quickly identify that a given file is code-generated. You could name the file with a "*.CodeGen.cs" extension, put a comment disclaimer at the top ("//This file is code-generated. Any changes will be overwritten"), and not check the target code into source control so that it doesn't have any overlay icons (like what SVN offers).
8. Know your overwrite strategy - If the target file already exists (because you're actively re-generating), make sure you know the expected behavior. If you don't use merge regions, you can simply overwrite the file. Source control should be smart enough to see that the file has the exact same content, and hence it shouldn't be a burden. Worst case, you can have your generator, before it writes out the generated context, detect if the target is the same or not, and handle appropriately (not write anything, have your build server throw a synchronization error if they're different, etc...)
9. Don't output the DateTime or user info into the code - When someone first uses a code generator, it can be tempting to add as much "free" details into the target file - like displaying "//This code was generated by Homer, on August 20,2009 at 11:34 pm". You'd never maintain that by hand, so it initially looks cool to see all that crisp information in your file. However, the problem is that every time the code is regenerated, that kind of information changes, and the code continually appears to be updated. Furthermore, such comments don't give you anything that you couldn't already get from source control.
10. Have a backup developer - Make sure that at least one other developer on the team can use the generating tool.

 

When should you use Code Generation?

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/when_should_you_use_code_generation.htm]

Like everything, using code generation is a tradeoff. Basically, you want codeGen when the cost of writing the templates is less than the costs of writing what the template generates. For example, codeGen rocks at creating and maintaining tedious plumbing code - data access plumbing is the canonical example.

You should probably consider code generation when the target code:

• Has similar patterns and deterministic rules, and little deviation from those rules. If there's an MS word doc or wiki page providing detailed instructions on how to write certain code, you may be able to send those instructions to the code generator instead.
• Is large and brittle, and it can be described easily (i.e. 1 line in an xml config fiile to describe = 20 lines of C# and SQL coding).
• Requires syncing with external data sources (like ADO.Net plumbing based on the database schema, such as var type, size, parameter order, count).
• Requires multiple files to be kept in sync (like a SQL stored proc being in sync with C# ADO.Net wrapper).
• Requires in-depth knowledge of a problem domain (like ADO.Net for data-access plumbing).
• Is continually updated (like adding new classes to your DAL).
• May need to be expanded in the future (like adding a whole new layer of webservices, or Audit triggers, to your DAL. Or, perhaps even migrating to a new language).

Code Generation is not a "Golden Hammer" - while it's great, it's not the perfect solution for everything. codeGen may not be the best solution if the target code:

• Is very custom with no general pattern. If you can't abstract a pattern out of the target code, then you won't be able to write a generation template.
• Is too small and trivial - In general, codeGen should decrease your total lines of code. So if you're writing a 50 line template to produce a single 30 line C# file, it's probably a bad ROI.
• Can be refactored away, and doesn't need to exist in the first place.

Like everything else, in some contexts, it just isn't profitable - but in other contexts it's awesome. Some problems are cheaper to solve using other techniques, like unit testing, automation, open-source, DSLs, or other techniques; but every advanced developer should have code generation in their tool belt.

Why code generate something after it is already written?

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/why_code_generate_something_after_it_is_already_written.htm]

If you've already written something, why go back and code-gen it? There could be a couple reasons:

• You're going to write a lot more of it, so you might as well leverage code-gen for all that existing code.
• You need to extend it with more functionality. Sure, the current stuff is already written, but you need to add new (tedious) features, like recording which properties of a class have had their setters called.
• You want to make it easier to maintain. Manually-maintained code counts as technical debt. Such code grows, get copied into off-by-one, and gradually gets customized. By code-generating it now, you'll make maintenance easier in the future.
• You want to document the rules. As codeGen essentially acts as documentation, this will give you a place to clearly define all those boundary cases.
• The code is already easy to generate. If the code is already prepared to be generated, might as well go ahead and migrate it.

Preparing your code to be code-generated

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/preparing_your_code_to_be_codegenerated.htm]

Sometimes you're not sure if something should be code-generated. If you do end up code-generating it in the future, there are several guidelines you can follow to make your life easier:

• Physically place the code in separate files. One of the hardest things to do with code-gen is to merge generated and custom code. By isolated the potential code in its own file, you'll spare yourself this pain.
• Standardize and simplify your code. For example, if you prefix all variables with Hungarian notation, your code-generator would need to do extra work to determine what that prefix is.
• Refactor as much as possible first. The less unique code you have, the easier it will be. there's also a good balance between code-generation and refactoring.
• Document the rules. If you're writing code based off a set of rules, document those rules - they'll come in handy as you try to generate the edge cases.

 

How to integrate Generated code into your application

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/how_to_integrate_generated_code_into_your_application.htm]

Code generation is great, but sometimes it can be confusing how to integrate that generated code into your custom application. Keep in mind that an application isn't solely SQL or C#. It could include Html, ASP.Net, Xml, JavaScript, project files, and much more.

• New file: Generate its own, new, separate file. This is the most basic way. You could then integrate it into your other files:
  • (C#) Base Class - For an OOP language like C#, you could code-generate the base class, and then have your derived classes inherit it.
  • (C#) Partial classes - Starting with .Net 2.0, C# offered partial classes which let you split the class definition across multiple physical files.
  • All: Include statements - Many languages offer a way to include one file within another. For example, ASP.Net offers server side includes, MSBuild offers the import command, HTML allows you to reference an external JavaScript, etc... (Yes, you could to this with SQL to using SqlCmds)
• Existing file: Merge into existing file with custom regions. For example, CodeSmith offers two kinds of custom regions:
  • InsertRegion - Insert your generated code into a marked region of a custom file
  • PreserveRegion - Insert your custom code into a code-generated file.

You could also integrate CodeSmith into your builds and processes by calling the CodeSmith console app.

Things to CodeGenerate besides the Data Access Layer

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/things_to_codegenerate_besides_the_data_access_layer.htm]

CodeSmith is a powerful code generator (worth its weight in gold). And while one of the most popular things to automatically generate is the data access layer, there's a lot more to CodeSmith than just wrapping databases:

• System data - Given an xml file, you can generate all your interrelated system data. For example, say your application has a data-driven relationship of groups, roles, menu items, tabs, and such (i.e. security and navigation) - you could write tons of tedious and brittle SQL scripts, or you could abstract it to an xml file and generate the SQL scripts from that.
• Data Structures - Especially before Generics in .Net 2.0, CodeSmith was popular for its strongly-typed collections (much faster performance than boxing and unboxing an ArrayList). You could make other data structures as well, depending on your application's need.
• Documentation - While CodeSmith's default DataDictionary template is popular for documenting your database schema, you could use CodeSmith as a Super-XSLT to transform any arbitrary xml list (like a file containing business rules, config, or test cases), into human-friendly HTML reports.
• Domain-Specific-Language - It's often more efficient to work at a higher level of abstraction. So, you could write an xml script, and use CodeSmith to translate that ("compile?") into useful actions.
  • Say you were trying to write automated UI tests, but the UI technologies keep changing, so you write a simple abstract xml script for the basic actions you care about (Load page, click button, etc...), and CodeSmith transforms that into the UI testing code for the relevant testing framework.
  • You could write abstract tests in xml (i.e. the data for pairs of input and output), and then use CodeSmith to dynamically generate all the unit tests from that.
  • You could read your file system to create an MSI installer using something like Wix.
• Starting Templates - I favor active re-generation when possible, and there's a balance between what to code-generate vs. what to refactor, however, sometimes it's useful to passively generate a starting template - just to give you a head start. For example, say your UI is too complicated to actively re-generate, but you could take an xml file of input and generate a stating template, from which you could then modify.

Basically, CodeSmith lets you take any input (a database, xml file, your file system, etc...) and generate any text output (sql, xml files, C#, aspx, html, js, etc...), and then also call C# to do anything on those files (install them in the database, commit it to source control, execute the resulting C#, etc...). It's a beautiful thing.

LCNUG: CodeSmith

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/lcnug_codesmith.htm]

We had our 9th LCNUG meeting yesterday. Dan Souk of Centric Consulting presented on CodeSmith, a topic near and dear to my heart:

• Using CodeSmith to create your own Domain-Specific-Language
• Code Generation templates act as your Documentation
• CodeSmith CodeGeneration: Super XSLT
• CodeSmith: Beyond Just Generation - Doing Simple Data Analysis
• Codesmith: Code Generation v. Refactoring
• Batch code generation with CodeSmith Console

Almost 20 people showed up, including several new folks. It's exciting to see the LCNUG continually grow.

Code Generation templates act as your Documentation

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/code_generation_templates_act_as_your_documentation.htm]

On every project, there's an endless list of things you're "supposed" to do, like documentation. I've personally never met a developer who enjoys this, so it usually gets shuffled to the side. One way to simplify documentation is to code generate what you can, because then the code generation template becomes your documentation.

Here's an example: I once was on a project where we had a large amount of base SQL data - things like populating dropdowns, security info, and config settings. Initially this was all done with hard-coded SQL files using tons of insert and update statements. Of course, to explain how to write all those inserts and updates, which tables they should alter, and what values they should set, there was tons of tedious MS word docs. Developers would constantly write failing SQL scripts, then recheck the documentation, fix their scripts, and keep trying until it worked. People always had questions like "Should the 'CodeId' column be set to the primary key from the 'RoleLookup' table? Technically, it "worked", it was just slow, brittle, and costing the business money.

Then, we tried migrating all those scripts over to xml files that got actively code generated into SQL scripts. All of a sudden, in order to actually write those templates, we needed to understand what the rules were. What tables, columns, and rows should be affected? What does an attribute like "active=true" translate into for a SQL script? So, this forced us to flush out all the rules. As time went on, when developers had questions, they could see all the rules "documented" in the code of the generation template (the *.cst file in CodeSmith). People essentially would say: "What happens if I set the entityScope attribute to zero? Oh, the code generator template specifically checks for that and will create this special insert in TableX when that happens. Ah, I see..."

Of course there are many other benefits of code generation, but I think this is an often overlooked one: generating the code from an xml file forces you to understand what you're actually doing. If you need to "explain" how to write code to a code generator, then anyone else can look at that explanation too. This is a similar principle to writing self-documenting code.

This also hit home after I wrote a UI page code generator that was 3000 lines long. I realized that in order for  a developer to create a UI page for that project, they essentially needed to (manually) reproduce whatever that template was doing, which meant that they had to understand 3000 lines of trivia. Anyone with questions on producing basic UI pages could always "look up" how the code generator did it.

Using CodeSmith to create your own Domain-Specific-Language

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/using_codesmith_to_create_your_own_domainspecificlanguage.htm]

Using CodeSmith to create your own Domain-Specific-Language

Yesterday I mentioned about software factories and domain-specific languages (DSL). A DSL is just that - working at a higher level language that maps to a specific problem domain instead of constantly re-inventing the wheel with a lower-level language. Some common examples of DSLs are:

• SQL
• Regular Expressions
• XPath

Each of these could be achieved by coding in a "low level" language like C#, but you wouldn't think of doing that because it'd be too slow and error prone. It's so easy to use each language because it maps naturally to what you're trying to do in that domain.

This same concept applies to application development. For example, an application has different domains:

• Initial data for your application (like security settings, roles, out-of-the-box dropdown values, etc...) - usually achieved with lots of custom SQL scripts.
• UI formatting - usually achieved with tons of table or CSS references, or highly-refactored controls
• Validation
• Data access code

Each of these can have their own DSL, which you could easily create using a code-generator like CodeSmith. You could abstract the concepts to an XML schema, and then use CodeSmith as a "Compiler" to transform that xml into the appropriate output (sql, html, or C# code files). CodeSmith's out-of-the-box XmlProperty feature, along with text based templates and huge online community make it very easy to do.

For example, instead of having tons of custom SQL scripts for your security data, you may have a hierarchal XML file that (1) is completely refactored and maps directly to the business needs (something potentially impossible with a procedural language like SQL), (2) can be easily validated via the XML schema and CodeSmith checks, and (3) is much easier to track version history on.

Microsoft offers their own DSL toolkit, but I think it doesn't yet compete with CodeSmith because the MS DSL toolkit: (1) requires you to learn a whole new GUI syntax (whereas CodeSmith is intuitive C# and Xml templates), (2) seems limited in what it can generate, and (3) screws up Visual Studio by inserting a new reference (or something like that) into every project.

Once you start code-generating tedious stuff, such as using your own DSL, you'll never go back.

---

Living in Chicago and interested in a great company? Check out the careers at Paylocity.

CodeSmith CodeGeneration: Super XSLT

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/codesmith_codegeneration_super_xslt.htm]

It's very common to have an XML data file, and then based on that to generate something else. Several years ago EXtensible Stylesheet Language Transformation (XSLT) was developed to handle this. The intent is that given an initial file, XSLT lets you program a template to transform that file into something else. For example, you could store just the raw data in an Xml file, and then have an XSLT template transform that into an html file. This new file is usually bigger and more redundant because it's doing something with the data, like adding extra html to format and present it. Another example would be storing base data in an Xml config file, and then transforming it to a lengthy SQL script.

The problem is that XSLT requires a entirely new skillset, isn't as powerful as a full-fledged language (like C#), and lacks the supporting development tools and IDE that make writing and debugging easier. You could use C# and the System.Xml namespace to manually transform the file, but that's tedious.

Another solution is to use a code-generation tool like CodeSmith. CodeSmith can take an Xml document, and given its XSD Schema, create a strongly typed object (which includes intellisense and collection capabilities). You can program a template in the CodeSmith IDE using C#. It's essentially like Super XSLT. By letting you use a language that you already know (either C# or VB), creating the strongly typed object, and using a standard IDE with debugging, using CodeSmith for batch (i.e. compile time) xml transformations can be a real time saver. It's also probably much easier for other developers to read and maintain a C#-based template than an XSLT one.

CodeSmith: Beyond Just Generation - Doing Simple Data Analysis

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/codesmith_beyond_just_generation__doing_simple_data_analysis.htm]

I've made several CodeSmith related posts on this blog:

• Codesmith: Code Generation v. Refactoring
• Batch code generation with CodeSmith Console
• Things that fundamentally change how you program

There's another topic I'd like to explore - using CodeSmith for analyzing  Xml data and Database schemas. For example, suppose you wanted to compare if two tables had similar schemas. For example I wanted to see if one database had different column lengths than another for all tables in the database. There's several ways you can do this:

• Manual --> bad (too slow, error prone, tedious, not-reproducible)
• Build your own custom console app, use ADO.Net --> very slow to create, to easy to have errors
• Find a custom tool specifically for DB comparisons --> yet another tool to learn, may provide far more functionality than needed
• Use CodeSmith's Schema Explorer --> has potential!

CodeSmith's Schema explorer is extensive - it lets you start at the database and drill down to its collections of tables, columns, and individual properties. So it has the info needed for most tasks. It's also easy to use, so it saves you time in writing your own equivalent object. It's also a reusable tool. I.e. if you need to learn a technology, learn one that is helps out with more than just the current problem.  You could create two SchemaExplorer.DatabaseSchema objects, and then cycle through their tables and columns, comparing them.

CodeSmith also has the XmlProperty, which lets you analyze an Xml document. You could program compare the Xml doc to the database schema.

The "generated code" could be the results of your analysis. For example, wherever the condition you're searching for is met, you could Response.WriteLine the necessary info.

While CodeSmith is designed for CodeGeneration, it also can be used for simple analysis of data sources that are commonly used as inputs for that generate - such as Xml docs and Database schemas.

Codesmith: Code Generation v. Refactoring

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/codesmith_code_generation_v_refactoring.htm]

Looking into CodeSmith, a good code generation tool, I started analyzing the differences between code generation and refactoring. Both help reduce redundancy. They are complimentary, with each concept having a different purpose:

• Code Generation - Given input values and a template, automatically generate redundant code that varies only by the input values. This deals with how you create code.
• Refactoring - Keep the functionality of your code the same, but improve the code itself via eliminating redundancy, improving clarity, etc... This deals with the final code, regardless of how that code was created.

Whenever you find yourself making repetitive code, you should at least check if these techniques can bail you out. Common examples may be (depending on your project) the data-access layer, the structure of business entities, documentation, unit tests, etc... Note that code generation and refactoring aren't opposites - but rather complementary. You can create a refactored class that still has some repetitiveness, and then code-generate it. For example, a fully refactored business entity may still have many properties that all follow the same format. For example, say your class has 10 such properties below. Given only the DataType and PropertyName, a code-generation template could crank this out:

public Double MyValue {     get     {         return _dblMyValue;     }     set     {         _dblMyValue = value;     } }

There are certainly places where repetitive code can be refactored such that there is no longer any redundancy, and therefore nothing to code-generate. A good code-generation tool is no excuse for not refactoring properly. However there are many places where refactoring alone is insufficient:

1. Design time: Code generation is done at design-time, and thus offers a performance benefit over refactoring (executed at run time). Say you needed a class to access the properties of your business entities. You could use reflection at design time, or (assuming you know the entities) you could use code-generation to create the necessary code before hand.
2. More functionality: Certain things can't effectively be refactored. If you needed to refactor a class that could handle many different Data Types, you'd need to use boxing./unboxing to handle the type-conversion. For example, pre-.Net 2.0 (which has generics) most collections, like HashTable or ArrayList, are not strongly typed. This requires boxing and unboxing which has obvious performance problems. .Net 1.1 does not provide a way to refactor the handling of various data types without the performance loss of boxing/unboxing. However you could use CodeSmith's strongly typed collection templates to automatically generate your own collections that don't require boxing. This is essentially still refactored because it provides additional functionality that the "refactored" version didn't meet - type safety.
3. Beyond source code: A good code generation tool, like CodeSmith, can create any text file, not just object-oriented source code. For example, suppose you wanted to document your database schema. CodeSmith provides pre-packaged templates that do this. I am not a aware of a way that "refactoring" would solve this problem.
4. Always automatic: Refactoring is great for object-oriented code with supporting unit tests. But it is difficult for many things outside of source code like documents or html pages. Short of other tools, these require a more manual approach. Code generation remains automatic

In conclusion, both refactoring and code generation are good things, but they are different and complimentary things.

Batch code generation with CodeSmith Console

[This was originally posted at http://timstall.dotnetdevelopersjournal.com/batch_code_generation_with_codesmith_console.htm]

I'm always keeping my eyes open for practical tools that save time and mental sanity. One of the most mentally-draining tasks is repetitious modifying of simple code. For example, many simple CRUD (Create-Read-Update-Delete) stored procedures, and their accompanying DAL and Business Objects are redundant. Ideally we could just auto-generate this given the inputs. Auto code generation is a great time saver, which frees us from boring tasks to focus on interesting ones.

CodeSmith is a great code generator. It has both a free and purchase version, it has a large support community, is time tested (been out for several years now), and has great functionality. It's based around writing templates with ASP syntax, which is much more friendly then using XSL or writing your own C# app to assemble the target code. Therefore part of a CodeSmith template may look like:

private void <%= SampleStringProperty %>() {     // <%= GetSomething() %>     // Do something }

There are plenty of tutorials about auto-generating your data access layer. But one thing I really like about CodeSmith is its batch processing. These are features I'd like in batch processing:

• Ability to call the tool from a command line, passing in an Xml config file
• Extend step 1 to generate many objects from a single Xml file, as opposed to calling the exe each time.
• Ability to merge your changes into an existing, non-generated document.
• Ability to auto-generate a document, but leave certain sections open to customization

The first two help with batch processing, such that you could generate an entire layer at once. The last two help with versioning and customization. You need to both made custom modifications to templates, and then not have those overwritten upon re-running the tool.

CodeSmith offers both several help guides, and samples, to assist with these. Check:

• CodeSmith Console Tutorial in the CodeSmith Studio's Help section
• Samples at: "CodeSmith\v3.0\SampleProjects\ConsoleSamples\GenerateCode.bat"

If all you have is a hammer, everything looks like a nail. With a good code-generation tool, you start viewing all those redundant tasks very differently.