I found this blog post about a couple Harvard students who wrote some [GPL'd] software for controlling worms' minds. They can control how these worms move and even make them lay eggs!
The implications of this are obviously huge. This is only an academic project now, but in a couple decades I wonder if we'll see animals used like machines? I guess there's several other ideas you could draw from this, but no matter how you view it, it's a fascinating idea.
Wednesday, January 19, 2011
Sunday, January 9, 2011
Declaring the Future of Programming
Programming languages have developed significantly over the past several decades. I hypothesize that this development has tended more towards declarative syntax than imperative. The future of programming languages will only become more declarative in the years to come.
In the beginning was machine code. Programmers wrote programs by stringing together arcane byte codes of instructions and parameters. Programs were getting pretty hard to read so they made assemblers so you could write instructions in plain text, complete with comments. An assembler program would process the source code and turn each instruction into it's equivalent machine code. This is imperative programming at its most pure state.
When the first C compiler was written it immediately became popular because the programmer only had to declare what should happen in the program and the compiler would generate the necessary machine code to make that happen. Hence why you can write a C program that can be compiled for Linux, Windows and Mac with zero changes to the source code. However, C and C++ are still imperative languages in most other aspects because the thought process is still very much a "do this, now do this, now do this" algorithmic sequence of instructions.
Query Languages
The hallmark of declarative languages thoughout history is probably SQL (referring strictly to set operations here). In SQL you describe the result set and let the DBMS decide the best way to produce that result set. For instance, consider this query:
First we describe the columns that we want (this actually happens last, if you want to be technical). In the from clause we say what tables we want information from and specify how we want them matched up using the on clause of the join. In the where clause we specify what criteria for the rows that we want to show and in the order by we describe the sort order.
All this was done strictly declaratively. If you have the opportunity to look at the execution plan, it all ends up being quite elaborate. It might consult two or three indexes before actually joining rows, selecting columns and ordering the result set not to mention all the asynchronous locking that took place so as not to run into race conditions. If we had to write this in C# or Java code it would be an extremely gnarly component and would probably be buggy and slow.
Expression Trees in C#
Interestingly, .NET land is also developing into a declarative playground. The biggest step in this direction happened with Linq and it's expression trees. Now, the Linq query syntax is declarative, but I'm referring to something more basic. Expression trees can be broken down at run time by a processor that can analyze the contents of a lambda that it was passed. For instance, NHibernate can receive a method call like:
and pull out the meaning (ResponsiblePerson = Tim) and convert it into a SQL "where" clause at run time (sql = "where a.ResponsiblePerson = 'Tim'). The implications of this are wild, and in recent months and years have become very powerful. Examples include Fluent NHibernate, Moq, and Castle Windsor's fluent registration API. Both castle windsor and NHibernate both used to use XML configuration files but have since moved towards using expression trees in combination with dynamic proxies and interceptors to configure via code. This declarative approach is leading towards less code that has potential to be more efficient.
Treatise on Domain Specific Languages
The topic of domain specific languages deserves an entire blog post. SQL and CSS are the obvious examples, but there are hundreds more. In one of my internships a coworker wrote a DSL to specify sort order for dictionaries for arcane natural languages and scripts. A simple DSL is much easier to develop than a GUI for the same purpose and can many times be easier for a non-techy user to learn and become productive in.
The sad news is that colleges and universities are putting less focus on compiler & parser classes. The assumption being that we have all the languages we need, why would we need more? The answer is simple: by providing a simple syntax to describe problems or solutions we can simplify the entire process of arriving to that solution. If the problem is abstracted away from the solution we can easily leverage constructs like multi-threading and highly optimized solutions. Sometime you should take a look at the byte codes that your compiler produces - ask yourself if you could have even thought of those sorts of mind bending tricks.
We need domain specific languages because they simplify problems. They create more effective abstraction than even inversion of control frameworks. Unfortunately, less people are learning about string processing these days. How many people have you worked with actually consider themselves proficient in regular expressions or compiler generators? (yet two more declarative DSLs that simplify solutions)
Conclusion
Anytime you write code that is less imperative, it allows the layer underneath more room to innovate efficient algorithms. Surely this isn't surprising since any good programmer would feel exactly the same way towards a micro-managing supervisor. So after saying all this, it should be clear why I believe that the future of programming is declarative. Declarative syntaxes allow us to simplify the problem by simply stating what the problem is (or describing what the solution looks like) and allowing the underlying engine to determine the solution. As such, I believe we will be seeing the number of domain specific languages multiply in the years to come.
In the beginning was machine code. Programmers wrote programs by stringing together arcane byte codes of instructions and parameters. Programs were getting pretty hard to read so they made assemblers so you could write instructions in plain text, complete with comments. An assembler program would process the source code and turn each instruction into it's equivalent machine code. This is imperative programming at its most pure state.
When the first C compiler was written it immediately became popular because the programmer only had to declare what should happen in the program and the compiler would generate the necessary machine code to make that happen. Hence why you can write a C program that can be compiled for Linux, Windows and Mac with zero changes to the source code. However, C and C++ are still imperative languages in most other aspects because the thought process is still very much a "do this, now do this, now do this" algorithmic sequence of instructions.
Query Languages
The hallmark of declarative languages thoughout history is probably SQL (referring strictly to set operations here). In SQL you describe the result set and let the DBMS decide the best way to produce that result set. For instance, consider this query:
select p.FirstName, p.LastName, a.AccountName from Person p inner join Account a on p.PersonId = a.ResponsiblePerson where a.IsActive = 1 order by p.FirstName, p.LastName
First we describe the columns that we want (this actually happens last, if you want to be technical). In the from clause we say what tables we want information from and specify how we want them matched up using the on clause of the join. In the where clause we specify what criteria for the rows that we want to show and in the order by we describe the sort order.
All this was done strictly declaratively. If you have the opportunity to look at the execution plan, it all ends up being quite elaborate. It might consult two or three indexes before actually joining rows, selecting columns and ordering the result set not to mention all the asynchronous locking that took place so as not to run into race conditions. If we had to write this in C# or Java code it would be an extremely gnarly component and would probably be buggy and slow.
Expression Trees in C#
Interestingly, .NET land is also developing into a declarative playground. The biggest step in this direction happened with Linq and it's expression trees. Now, the Linq query syntax is declarative, but I'm referring to something more basic. Expression trees can be broken down at run time by a processor that can analyze the contents of a lambda that it was passed. For instance, NHibernate can receive a method call like:
var timsAccounts = accounts.Where(x => x.ResponsiblePerson == "Tim");
and pull out the meaning (ResponsiblePerson = Tim) and convert it into a SQL "where" clause at run time (sql = "where a.ResponsiblePerson = 'Tim'). The implications of this are wild, and in recent months and years have become very powerful. Examples include Fluent NHibernate, Moq, and Castle Windsor's fluent registration API. Both castle windsor and NHibernate both used to use XML configuration files but have since moved towards using expression trees in combination with dynamic proxies and interceptors to configure via code. This declarative approach is leading towards less code that has potential to be more efficient.
Treatise on Domain Specific Languages
The topic of domain specific languages deserves an entire blog post. SQL and CSS are the obvious examples, but there are hundreds more. In one of my internships a coworker wrote a DSL to specify sort order for dictionaries for arcane natural languages and scripts. A simple DSL is much easier to develop than a GUI for the same purpose and can many times be easier for a non-techy user to learn and become productive in.
The sad news is that colleges and universities are putting less focus on compiler & parser classes. The assumption being that we have all the languages we need, why would we need more? The answer is simple: by providing a simple syntax to describe problems or solutions we can simplify the entire process of arriving to that solution. If the problem is abstracted away from the solution we can easily leverage constructs like multi-threading and highly optimized solutions. Sometime you should take a look at the byte codes that your compiler produces - ask yourself if you could have even thought of those sorts of mind bending tricks.
We need domain specific languages because they simplify problems. They create more effective abstraction than even inversion of control frameworks. Unfortunately, less people are learning about string processing these days. How many people have you worked with actually consider themselves proficient in regular expressions or compiler generators? (yet two more declarative DSLs that simplify solutions)
Conclusion
Anytime you write code that is less imperative, it allows the layer underneath more room to innovate efficient algorithms. Surely this isn't surprising since any good programmer would feel exactly the same way towards a micro-managing supervisor. So after saying all this, it should be clear why I believe that the future of programming is declarative. Declarative syntaxes allow us to simplify the problem by simply stating what the problem is (or describing what the solution looks like) and allowing the underlying engine to determine the solution. As such, I believe we will be seeing the number of domain specific languages multiply in the years to come.
Sunday, January 2, 2011
Would I choose Git again?
I wrote a post a few months ago about the reasons we chose to use Git over subversion and I think it's time to follow up that post and write about how its gone so far. We're an ASP.NET outfit, and as such there are a few considerations that might not apply to, say, the Linux kernel team. I'm going to break this up into three parts: my perspective, my team's perspective, and some tips for anyone who might want to also try using Git.
My Experiences With Git
I seriously love using Git. I make a branch for everything I do just like they recommend. An old-school member of our team made a comment, "we always considered branches as something to be avoided", hinting at SVN branches' trait of being hard to manage and keep in sync with the trunk. Git branches are very different from SVN branches - they are very light and easy to keep up to date.
Git has some seriously awesome merging mechanisms. First, you can select from a list of merge algorithms (you really only need one of these, but hey, its great to have choices just in case). Then they also have rebase and cherry-picking. These last two aren't regular merges because their algorithms look at the history of the entire repository and make several [and possibly hundreds of] incremental merges. Because these schemes take history into account, you can actually do some serious refactoring and still apply patches to both the production and development branches with relatively little effort.
Our team develops and maintains a web application that our company sells as a service. As such, we don't spend time on installers or maintaining previous versions because the only versions that matter are the version that's in production and the development version. Git allows us to cherry-pick hotfixes from development into production (or vice versa) without really thinking much. This would have been a small nightmare in SVN (and invoke suicidal tendencies in TFS). Back when we were using TFS there really wasn't any process or procedure that went into hotfixes. You basically just updated production. With Git, its incredibly easy to just stash whatever you're doing, checkout the production branch, fix a critical bug, test & deploy it, an then cherry pick it back into the dev branch. Git works well for people who get interrupted by escalations (everyone??).
My Team's Experiences
My team hates Git. Well, that's a bit harsh and premature, but there was some backlash when we first switched. About three weeks in I gave a brown bag lunch presentation on Git to teach everyone how to use it. After that people generally caught on to the basics with exception of some merging snafus.
Merging is actually an interesting point. TFS merging drove me nuts. Perhaps it was just the merge program, but I always felt like I had my hands tied. Now that we're using Git I feel free again to branch and merge at will, but one of my teammates seemed to be (at least at first) completely confused by Git merging. This was [probably] entirely due to the fact that Git Extensions didn't come with kdiff by default (they now offer a convenient all-in-one installer that includes kdiff & Git).
Another point of confusion in using Git GUIs was that TortoiseGit makes it very difficult to see what's different between local and remote repositories. I think the Tortoise crew made too much of an effort to make it feel like TortoseSVN when in reality it left some very important questions unanswered (TortoiseSVN only has to answer 1 or 2 important questions, but Git GUIs need to answer 4 or 5 important questions). Among these unanswered questions are "what branch am I on?" and "have I pushed this to the server yet?". TortoiseGit doesn't provide a clear answer to either of these questions, so I had everyone make a switch to Git Extensions.
Tips for Future Git Users
We were forced to learn a few lessons pretty quickly. I'll list them here in paragraph format...
GUIs are still young. Most Git users are sick Linux users who live by vi & grep, so developing a decent GUI hasn't really been a priority for Git (there is an official Git GUI that ships with Git, but it possesses some serious suckage). If you work in a Microsoft/Windows outfit there is no conceivable way your coworkers will be happy with command line, so a good GUI is critical. Use Git Extensions!
Setting up a central server is not entirely straightforward. While SVN is distributed as either a client or a server, Git has no reason to require a central server so this was also an afterthought. Use gitolite on Linux. Use the package manager method of installing it, its very easy to get it started and its also easy to maintain.
SSH keys are problematic. Try to use putty/plink to manage keys if possible. OpenSSH is very un-Windows-like.
Unit tests are good and they can make Git shine even brighter. If you maintain a generally complete unit test suite you can have Git utilize your test runner to quickly find where code started breaking. The "bisect" command can take a program or command that returns 0 or 1 (standard success/failure codes, so throwing exceptions would work) and perform a binary search through past commits to find the first place where a test started failing. This could also work great if you're a scripting guru - write a short script to check for some text (like "CREATE TABLE X") in a particular file and Git will do the leg work.
Conclusive Thoughts
Git is very powerful and can adapt to any workflow. If process is important to you, Git will enable you in whatever process you choose. If process isn't important, Git won't get in your way. It is very scalable via its distributed nature (ref dictator and lieutenants). It's also great for small personal projects that I do in my spare time. I can still have code version controlled without sharing it with anyone, but when I want to I can push it to Github (another awesome idea). However, if your coworkers are generally stagnant and opposed to change, Git will drive them nuts and you will hate your life. Choose Git only if you want a program that will abstract away mundane tasks like merging but you don't mind having to change your world view towards version control.
My Experiences With Git
I seriously love using Git. I make a branch for everything I do just like they recommend. An old-school member of our team made a comment, "we always considered branches as something to be avoided", hinting at SVN branches' trait of being hard to manage and keep in sync with the trunk. Git branches are very different from SVN branches - they are very light and easy to keep up to date.
Git has some seriously awesome merging mechanisms. First, you can select from a list of merge algorithms (you really only need one of these, but hey, its great to have choices just in case). Then they also have rebase and cherry-picking. These last two aren't regular merges because their algorithms look at the history of the entire repository and make several [and possibly hundreds of] incremental merges. Because these schemes take history into account, you can actually do some serious refactoring and still apply patches to both the production and development branches with relatively little effort.
Our team develops and maintains a web application that our company sells as a service. As such, we don't spend time on installers or maintaining previous versions because the only versions that matter are the version that's in production and the development version. Git allows us to cherry-pick hotfixes from development into production (or vice versa) without really thinking much. This would have been a small nightmare in SVN (and invoke suicidal tendencies in TFS). Back when we were using TFS there really wasn't any process or procedure that went into hotfixes. You basically just updated production. With Git, its incredibly easy to just stash whatever you're doing, checkout the production branch, fix a critical bug, test & deploy it, an then cherry pick it back into the dev branch. Git works well for people who get interrupted by escalations (everyone??).
My Team's Experiences
My team hates Git. Well, that's a bit harsh and premature, but there was some backlash when we first switched. About three weeks in I gave a brown bag lunch presentation on Git to teach everyone how to use it. After that people generally caught on to the basics with exception of some merging snafus.
Merging is actually an interesting point. TFS merging drove me nuts. Perhaps it was just the merge program, but I always felt like I had my hands tied. Now that we're using Git I feel free again to branch and merge at will, but one of my teammates seemed to be (at least at first) completely confused by Git merging. This was [probably] entirely due to the fact that Git Extensions didn't come with kdiff by default (they now offer a convenient all-in-one installer that includes kdiff & Git).
Another point of confusion in using Git GUIs was that TortoiseGit makes it very difficult to see what's different between local and remote repositories. I think the Tortoise crew made too much of an effort to make it feel like TortoseSVN when in reality it left some very important questions unanswered (TortoiseSVN only has to answer 1 or 2 important questions, but Git GUIs need to answer 4 or 5 important questions). Among these unanswered questions are "what branch am I on?" and "have I pushed this to the server yet?". TortoiseGit doesn't provide a clear answer to either of these questions, so I had everyone make a switch to Git Extensions.
Tips for Future Git Users
We were forced to learn a few lessons pretty quickly. I'll list them here in paragraph format...
GUIs are still young. Most Git users are sick Linux users who live by vi & grep, so developing a decent GUI hasn't really been a priority for Git (there is an official Git GUI that ships with Git, but it possesses some serious suckage). If you work in a Microsoft/Windows outfit there is no conceivable way your coworkers will be happy with command line, so a good GUI is critical. Use Git Extensions!
Setting up a central server is not entirely straightforward. While SVN is distributed as either a client or a server, Git has no reason to require a central server so this was also an afterthought. Use gitolite on Linux. Use the package manager method of installing it, its very easy to get it started and its also easy to maintain.
SSH keys are problematic. Try to use putty/plink to manage keys if possible. OpenSSH is very un-Windows-like.
Unit tests are good and they can make Git shine even brighter. If you maintain a generally complete unit test suite you can have Git utilize your test runner to quickly find where code started breaking. The "bisect" command can take a program or command that returns 0 or 1 (standard success/failure codes, so throwing exceptions would work) and perform a binary search through past commits to find the first place where a test started failing. This could also work great if you're a scripting guru - write a short script to check for some text (like "CREATE TABLE X") in a particular file and Git will do the leg work.
Conclusive Thoughts
Git is very powerful and can adapt to any workflow. If process is important to you, Git will enable you in whatever process you choose. If process isn't important, Git won't get in your way. It is very scalable via its distributed nature (ref dictator and lieutenants). It's also great for small personal projects that I do in my spare time. I can still have code version controlled without sharing it with anyone, but when I want to I can push it to Github (another awesome idea). However, if your coworkers are generally stagnant and opposed to change, Git will drive them nuts and you will hate your life. Choose Git only if you want a program that will abstract away mundane tasks like merging but you don't mind having to change your world view towards version control.
Thursday, November 4, 2010
Why Linux Sucks
Just to be clear, I have had Linux on my main home computer for several years. In fact, I'm writing this on Linux and I'm not having any problems. I have no intention of giving up Linux. I like how it works and I like tinkering with the different parts of it.
I use Ubuntu. Ubuntu really is Linux for humans - its easy to use and everything just works. Well...almost everything. I installed the 64-bit version and Adobe didn't support 64-bit flash for a long time (and I couldn't install 32-bit Firefox). Seriously, how many web sites use flash? Essentially every site that my wife and I both use. My wife hates Linux.
There's two sides to the Linux community. There are the people who want to see Linux for the masses (Canonical & team) and then there's the hardcore users.
The thing that really gets me about Linux is that the hardcore users have no intention of making Linux easier to use. I usually don't have a problem finding Linux help on the Internet, but the gurus that answer Linux questions aren't particularly easy going. I've spent enough time reading through forums for Linux help that I know that they follow a strict rubric:
I use Ubuntu. Ubuntu really is Linux for humans - its easy to use and everything just works. Well...almost everything. I installed the 64-bit version and Adobe didn't support 64-bit flash for a long time (and I couldn't install 32-bit Firefox). Seriously, how many web sites use flash? Essentially every site that my wife and I both use. My wife hates Linux.
There's two sides to the Linux community. There are the people who want to see Linux for the masses (Canonical & team) and then there's the hardcore users.
The thing that really gets me about Linux is that the hardcore users have no intention of making Linux easier to use. I usually don't have a problem finding Linux help on the Internet, but the gurus that answer Linux questions aren't particularly easy going. I've spent enough time reading through forums for Linux help that I know that they follow a strict rubric:
- Always use command line. The biggest thing is installing new programs and packages. They could easily tell someone that they need to install package x, but instead they always use the command line:
sudo apt-get install destroy_linux
Seriously, why can't you just use the pretty UI that Ubuntu created for installing software? I know they are easy commands, but seriously. Not making things easy for my wife. - Always make things more complicated than necessary. Usually this involves using the command line with three times as many commands than you really need. But also chastising for silly questions
- Keep things magical. Magical lands are fun at Disney land, but I hate punching in inexplicably terse text into a console. The terms and commands become shorter and less descriptive as you get deeper into Linux (there is no end). Don't try to understand.
Tuesday, October 19, 2010
Object-Form mapping
I'm pretty sure most developers (web developers anyway) have heard of ORM (Object Relational Mapping) tools like NHibernate that map your database tables and to objects. These ORM tools reduce interaction with the database to just a few method calls, many times just Save(), GetById(), and a few custom query methods. There's a lot written about ORM, but no one really writes about the mapping between HTML forms and the objects that ORM maps.
ASP.NET has a great solution for OFM (I'm calling it OFM because google won't give me a real name for it). If you use a FormView in combination with an ObjectDataSource you can bind the properties of your object to form elements. This is pretty cool because it reduces your code to writing an ORM mapping, creating factory methods to get and save the object, and some ASP markup that maps the object to HTML elements.
I was playing with Ruby on Rails which has a somewhat different approach to OFM. Basically you write regular HTML and give your form elements names like "account[id]", "account[name]", etc. This seems like a little more work than the ASP.NET way except that on the server side it uses this notation to wrap the query string into an object that can be referenced in object notation from ruby code like "account.id", "account.name", etc. I believe PHP does something similar. I like this method because it's very light on HTTP - there's no obstructively bloated view state being passed around like there is in ASP.NET and you can pass several objects through the query string.
Basically, OFM manages some of the page flow by marshalling form parameters into objects that can easily be passed to a factory method. This is awesome because it means I can focus more effort on writing unit tests for business logic that has no dependencies on the web API. It allows me to to keep page flow simple and sets up business logic for creating restful web services (seriously, you could just slap [WebMethod] attributes on the factory methods and voila you have web services). There seems to be a lot of framework that goes into managing OFM, but oddly I don't think many people have addressed it directly as a problem that needs to be overcome (I assume this is because the MVC architecture is supposed to address this; unfortunately vanilla ASP.NET isn't MVC).
I recently pulled most of my hair out over the ObjectDataSource and interfacing with factory methods. In the future I want to write a post about how I got around it (and another one lambasting Microsoft for even attempting to release an API as thoughtless as the ODS, but seriously, more on that later).
ASP.NET has a great solution for OFM (I'm calling it OFM because google won't give me a real name for it). If you use a FormView in combination with an ObjectDataSource you can bind the properties of your object to form elements. This is pretty cool because it reduces your code to writing an ORM mapping, creating factory methods to get and save the object, and some ASP markup that maps the object to HTML elements.
I was playing with Ruby on Rails which has a somewhat different approach to OFM. Basically you write regular HTML and give your form elements names like "account[id]", "account[name]", etc. This seems like a little more work than the ASP.NET way except that on the server side it uses this notation to wrap the query string into an object that can be referenced in object notation from ruby code like "account.id", "account.name", etc. I believe PHP does something similar. I like this method because it's very light on HTTP - there's no obstructively bloated view state being passed around like there is in ASP.NET and you can pass several objects through the query string.
Basically, OFM manages some of the page flow by marshalling form parameters into objects that can easily be passed to a factory method. This is awesome because it means I can focus more effort on writing unit tests for business logic that has no dependencies on the web API. It allows me to to keep page flow simple and sets up business logic for creating restful web services (seriously, you could just slap [WebMethod] attributes on the factory methods and voila you have web services). There seems to be a lot of framework that goes into managing OFM, but oddly I don't think many people have addressed it directly as a problem that needs to be overcome (I assume this is because the MVC architecture is supposed to address this; unfortunately vanilla ASP.NET isn't MVC).
I recently pulled most of my hair out over the ObjectDataSource and interfacing with factory methods. In the future I want to write a post about how I got around it (and another one lambasting Microsoft for even attempting to release an API as thoughtless as the ODS, but seriously, more on that later).
Tuesday, October 12, 2010
Why we chose Git instead of Subversion
I just got a new job as an ASP.NET developer at a small company that is freshly developing itself into somewhat of a software company. The development team is undergoing a ton of changes over the past 6 months (6 months ago there were two developers, now there is five as well as a new director of technology). As part of our changes we took some time to evaluate the tools we use. We had been using Microsoft's Team Foundation Server for source control and a home-grown system for bug tracking but after our evaluations we settled on Redmine and Git.
The fact that we are using Redmine for ALM and bug tracking isn't particularly surprising to me because it's a feature heavy and mature product that is very natural to use. There are several other feature heavy mature ALM tools that would fit us, but none that are free (I don't consider Trac feature heavy). Git, however, is a bit of a pleasant surprise for me.
For the uninitiated ones, Git is a distributed SCM (source control management) tool. The distributed part means that it works kind of like Subversion except that everyone has a full clone of the repository. When you want to check your code in you commit first to yourself and then push your changes to the rest of the team. More realistically you would be committing to yourself several times and occasionally pushing your changes to the rest of the team when you verify that your code is stable.
The benefit of this is that you can maintain your own personal branches of the code where you experiment on certain features without having to push them out to everyone else. I see this as psychologically breaking down the barrier to committing code. I often find that I don't commit code for a while because, even though it builds, I'm not sure if some of the pages will run without errors. However, committing to myself means that I can commit whenever I want and not slow any of my teammates down with potential errors.
Git also provides very easy and simple branching. They made it extremely easy to drop everything your doing to fix that top priority bug in production (the "stash" operation lets you save uncommitted changes and move to another part of the code). With this extra change management, Git also forces you to account for all your changes. Before you switch branches you have to either stash, commit or revert your current changes. At first this seems annoying, but on second thought it forces to always have some sort of accounting for why you changed stuff.
We did have some hesitation with changing to Git. Our biggest concern was if one of our partner teams from a different company could keep up with a change in SCM. After some evaluation we realized that Git provided so much flexibility with managing our workflow with this partner that it makes Subversion look like an archaic hack.
Another concern we had was stability. Git itself has been around since 2005 and seems to have pretty strong development community backing it. It has a very strong Linux following and a year ago lacked a good Windows interface. However, TortoiseGit has been developing at a very rapid rate (it's single developer has been releasing more than twice a month and is quickly working toward supporting most of Git's features). Because it is developing so fast we agreed that we could disregard shortcomings in the Windows environment in due to the awesome number and power of the features it brings.
Today I worked on importing our TFS repository into a Git clone. I found a PowerShell script hosted on Github that got me pretty close. The code in the script was a little too brittle so I made the code a little more generic and sent it back to him. It's taking about six hours to migrate the 1200 changesets into Git, so the script probably won't finish running for another couple hours, but I think it's working so far.
I will have to follow up in six months or so with an evaluation of how things have gone.
The fact that we are using Redmine for ALM and bug tracking isn't particularly surprising to me because it's a feature heavy and mature product that is very natural to use. There are several other feature heavy mature ALM tools that would fit us, but none that are free (I don't consider Trac feature heavy). Git, however, is a bit of a pleasant surprise for me.
For the uninitiated ones, Git is a distributed SCM (source control management) tool. The distributed part means that it works kind of like Subversion except that everyone has a full clone of the repository. When you want to check your code in you commit first to yourself and then push your changes to the rest of the team. More realistically you would be committing to yourself several times and occasionally pushing your changes to the rest of the team when you verify that your code is stable.
The benefit of this is that you can maintain your own personal branches of the code where you experiment on certain features without having to push them out to everyone else. I see this as psychologically breaking down the barrier to committing code. I often find that I don't commit code for a while because, even though it builds, I'm not sure if some of the pages will run without errors. However, committing to myself means that I can commit whenever I want and not slow any of my teammates down with potential errors.
Git also provides very easy and simple branching. They made it extremely easy to drop everything your doing to fix that top priority bug in production (the "stash" operation lets you save uncommitted changes and move to another part of the code). With this extra change management, Git also forces you to account for all your changes. Before you switch branches you have to either stash, commit or revert your current changes. At first this seems annoying, but on second thought it forces to always have some sort of accounting for why you changed stuff.
We did have some hesitation with changing to Git. Our biggest concern was if one of our partner teams from a different company could keep up with a change in SCM. After some evaluation we realized that Git provided so much flexibility with managing our workflow with this partner that it makes Subversion look like an archaic hack.
Another concern we had was stability. Git itself has been around since 2005 and seems to have pretty strong development community backing it. It has a very strong Linux following and a year ago lacked a good Windows interface. However, TortoiseGit has been developing at a very rapid rate (it's single developer has been releasing more than twice a month and is quickly working toward supporting most of Git's features). Because it is developing so fast we agreed that we could disregard shortcomings in the Windows environment in due to the awesome number and power of the features it brings.
Today I worked on importing our TFS repository into a Git clone. I found a PowerShell script hosted on Github that got me pretty close. The code in the script was a little too brittle so I made the code a little more generic and sent it back to him. It's taking about six hours to migrate the 1200 changesets into Git, so the script probably won't finish running for another couple hours, but I think it's working so far.
I will have to follow up in six months or so with an evaluation of how things have gone.
Tuesday, June 8, 2010
CouchDB + Ext as a Replacement for Server Code
In a previous post about ExtJS I mentioned the possibility of developing a web application that runs entirely inside the browser and doesn't require any server side code. The idea stems from a) ExtJS is a fully capable widget framework and b) CouchDB is accessible via a web service. At least 80% of web apps are just a HTML interface with a database back-end and a little bit of business logic. So why can't we move all that business logic to the browser, setup calls to a CouchDB web service from the browser and 86 the server-side code? In this post I'm going to analyze this question and see if it's realistic. In a follow up post I'm going to analyze this same question from a business standpoint.
A Database Void of Schema
CouchDB is a document oriented database, meaning that it doesn't have tables and keys like you do in relational databases. It just has one big space full of documents. A document in CouchDB is a JSON object, so its attribute values can be strings, booleans, numbers, lists, or other objects (documents). Having complex "rows" means that many of your relationships that you would normally form by using a second table and a primary-foreign key set is simplified down to embedding a list. Consequently, 1-to-1 and 1-to-many relationships are native to the database and require no extra thought or planning. Many-to-many relationships are more complicated, so this approach might break down if you require too many of these. Some other oddities in relational databases like versioning and pivot tables come native with CouchDB. Since the bulk of our database requirements are made easier with CouchDB, querying is going to be generally simpler.
The other great thing about having a document formatted in JSON is that you can save any JavaScript object directly to the database. You could save the state of an Ext widget or a whole form. It's like simplified object serialization for the browser! This is definitely a killer argument for making fat client apps with Ext.
But What About Performance?
At some point, someone's going to ask it. I say, Twitter uses it, they seem to be doing well, there's one case that its proven itself. The biggest argument for CouchDB being a scalable database is the fact that it is built from the ground up with the intent of being distributed across many nodes in a cloud. So while it is easy to get a database stood up for development, it's just as easy to move that database into a highly distributed cloud with hundreds of nodes. This makes it easy to develop scalable world-class apps like Twitter or Google.
CouchDB uses a type of index that is based on the map-reduce algorithm used in functional programming. You define a function in JavaScript that takes a list of values and chooses which ones to include in a view. When you want to query the database you just ask it for all or part of a view. Because it uses the map-reduce algorithm to index, it's agnostic towards when and how many documents are indexed at a time. So documents can be quickly indexed on insertion/creation, or the whole database can be indexed at one time.
If the client code is developed entirely in Ext and JavaScript, all forms are static HTML pages, so the server can easily respond to 80% of requests with little more than a few HTTP headers (client-side caching).
What About Security?
At this point someone must be ready to blurt out something about this being an incredibly insecure approach to web development. After all, any slick hacker can modify the JavaScript code and execute arbitrary insertions/deletions. Security is definitely going to be a lot bigger of a concern in this case. However, CouchDB does provide fine grained security controls. Here is an informative video about CouchDB security controls.
The big difference with designing security into couch apps is that security is going to be built into the database instead of the application. CouchDB provides constructs for users to be part of roles. If constructed well, the developer can leverage the database to deny or allow certain operations for the current user.
Taking the Ext + CouchDB approach is going to be a fundamental shift in application design. If we learned to write apps like this we might actually learn to rely on the framework to do what it does best, and let our app do only what it needs to do. We might even find ourselves making stable and secure apps in less time.
Conclusion
From a technical perspective, I think this might be a very feasible design paradigm. In a coming post I am going to talk about the business costs involved. However, I think document oriented databases might be something I want to investigate further and design into future applications.
A Database Void of Schema
CouchDB is a document oriented database, meaning that it doesn't have tables and keys like you do in relational databases. It just has one big space full of documents. A document in CouchDB is a JSON object, so its attribute values can be strings, booleans, numbers, lists, or other objects (documents). Having complex "rows" means that many of your relationships that you would normally form by using a second table and a primary-foreign key set is simplified down to embedding a list. Consequently, 1-to-1 and 1-to-many relationships are native to the database and require no extra thought or planning. Many-to-many relationships are more complicated, so this approach might break down if you require too many of these. Some other oddities in relational databases like versioning and pivot tables come native with CouchDB. Since the bulk of our database requirements are made easier with CouchDB, querying is going to be generally simpler.
The other great thing about having a document formatted in JSON is that you can save any JavaScript object directly to the database. You could save the state of an Ext widget or a whole form. It's like simplified object serialization for the browser! This is definitely a killer argument for making fat client apps with Ext.
But What About Performance?
At some point, someone's going to ask it. I say, Twitter uses it, they seem to be doing well, there's one case that its proven itself. The biggest argument for CouchDB being a scalable database is the fact that it is built from the ground up with the intent of being distributed across many nodes in a cloud. So while it is easy to get a database stood up for development, it's just as easy to move that database into a highly distributed cloud with hundreds of nodes. This makes it easy to develop scalable world-class apps like Twitter or Google.
CouchDB uses a type of index that is based on the map-reduce algorithm used in functional programming. You define a function in JavaScript that takes a list of values and chooses which ones to include in a view. When you want to query the database you just ask it for all or part of a view. Because it uses the map-reduce algorithm to index, it's agnostic towards when and how many documents are indexed at a time. So documents can be quickly indexed on insertion/creation, or the whole database can be indexed at one time.
If the client code is developed entirely in Ext and JavaScript, all forms are static HTML pages, so the server can easily respond to 80% of requests with little more than a few HTTP headers (client-side caching).
What About Security?
At this point someone must be ready to blurt out something about this being an incredibly insecure approach to web development. After all, any slick hacker can modify the JavaScript code and execute arbitrary insertions/deletions. Security is definitely going to be a lot bigger of a concern in this case. However, CouchDB does provide fine grained security controls. Here is an informative video about CouchDB security controls.
The big difference with designing security into couch apps is that security is going to be built into the database instead of the application. CouchDB provides constructs for users to be part of roles. If constructed well, the developer can leverage the database to deny or allow certain operations for the current user.
Taking the Ext + CouchDB approach is going to be a fundamental shift in application design. If we learned to write apps like this we might actually learn to rely on the framework to do what it does best, and let our app do only what it needs to do. We might even find ourselves making stable and secure apps in less time.
Conclusion
From a technical perspective, I think this might be a very feasible design paradigm. In a coming post I am going to talk about the business costs involved. However, I think document oriented databases might be something I want to investigate further and design into future applications.
Subscribe to:
Posts (Atom)