The frugal programmer

In the current (2009) economic recession, many blogs about frugality have encountered their major success, aiding people to track and control their spending. A specific frugal approach for programmers and system administrators can help saving bucks, especially if you are exploring the freelancing world or bootstrapping a new business, and every dollar counts.

A side business can be an interesting adventure, and since many new business fail in the first years a little budget is the way to reduce risk to a fixed amount of money and time, playing with what you can afford to lose only. Since this strategy will result in a tight budget, here are some tips on how to manage your freelancer tools along with keeping an eye on how much you are spending.

• upgrade boxes with crucial hardware. Instead of purchasing a brand new pc, in many cases a pair of new parts (Ram and disk space) can extend the box life of some years; of course a lighter OS will contribute to the revitalization.
• recycle old machines. Lan servers do not have big requirements and a source control server can be built from nearly anything. I use a old laptop with broken lcd screen and automatic connection to the wireless lan as a http server for staging php applications without weighing down my development box.
• recycle peripherals from previous machines. Many pieces of equipment have follow the same standard for decades and can be ported from an old machine to empower a new one. Printers and scanners are obviously regained but also keyboard, mouse and monitor can be attached to a new box as they have no performance penalties.
• use Open Source Software. Ubuntu Linux is a very solid distribution and it contains many application for every kind of usage; I use only Ubuntu on my home pcs: Apache, Php, Mysql, Subversion are examples of what Open Source can do. The expertise that you learn using Open Source tools is also a good know-how that can be useful in your future jobs and projects.
• buy consumables in bulk. Faster networks and flash cards are killing compact discs and other old media, but when they are still useful you should buy them in large packages (50, 100 pieces) as the total cost can go down to half of the original.
• do not buy Apple. I think quality is why a Mac cost so much, but an Apple product comes with little upgrading capability, and with a stable but fixed Os where you have to use port of popular applications. I know Macs have nice curves and a good user experience but it does not help when you have to save money: for instance I mainly use command line and vim, where a Mac does not excel more than a old pc with Ubuntu Jaunty installed.

I hope these strategies can help you spend your money better. You should make investments in your tools by spending where you feel a return, such as a comfortable and well designed chair or a lcd monitor for preserving your eyes; only ensure you are not throwing money down the drain.

Applying Domain-Driven Design and Patterns review

Applying Domain-Driven Design and Patterns: With Examples in C# and .NETis a book from Jimmy Nilsson which shows you how to begin applying such things as TDD, object relational mapping, and DDD to .NET projects...techniques that many developers think are the key to future software development. Although it is based on .NET technology and features C# examples, it is a valuable resource for DDD adopting in every object-oriented language and one of the few practical books on Domain-Driven Design, along with the original one from Eric Evans.

The main feature of this book is a full description of the process followed in development: previous solutions are presented along with improvements to the object-oriented design and the reasons that have lead to refactoring. For instance, persistence ignorance is not assumed as the basis of a decoupled model but it is the final point of a research path.
This is a panoramic of the topics which the author deals with in the 600+ pages book.

Part 1: Background
The first part of the book is a introduction on low-level techniques like TDD, refactoring, and application of patterns (for example, State pattern). Problem and tradeoffs are discusses regarding databases and their impedance mismatch with an object-oriented model, together with the importance of layering and modelling. Part 1 is important because understanding of the rest of the book strongly depends on these concepts: a responsible programmer should test first and refactor where he expects changes.

Part 2: Applying DDD
In part 2 a full application requirements are discussed along with part of their implementation. Since the title is applying DDD the focus is on building the right model for the example, leaving out infrastructure and security concerns for a moment.
The requirements list comprehends orders and customers management, concurrency conflict detection, validation rules on amount of moneys in orders and customers, atomic saving and validation with an external service. This list is well crafted as it contains the main problems encountered when using a rich Domain Model, so the order management application is a good example to see DDD applied at its best.

Part 3: Applying PoEaa
While part 2 is centered on a persistent-ignorant model, part 3 discuss the infrastructure needed for the example application to work (object/relational mapping in primis) and tradeoffs from the incorporation of database concerns in the pure domain model. PoEaa is Patterns of Enterprise Application Architecture,a famous book from Martin Fowler which presents the principal patterns used in orms and big applications.
PoEaa is full of code examples so it is not important for the author to attach naked code but the discussion on how to persist a rich model is still crucial.

Part 4: What's next?
This part introduces other techniques and topics which are not strictly related to modelling and infrastructure: Service-oriented architecture, Inversion of Control and Dependency Injection, Aspect-oriented programming, Model-View-Controller pattern, user interface development. These are all interesting topics which a smattering of can only help the average developer to enlighten himself and face different ideas and paradigms: TDD for instance needs Dependency Injection to work.

Part 5: Appendices
Last but not least, other styles are proposed in contrast to DDD, such as database-driven development and rich service layers. The same application requirements from part 3 are rediscussed using these new point of views, and the reader gets a comparison between various implementation of the Domain Model pattern.
The appendices contain also a list of the patterns referenced in the book, but the should consider other sources for more information.

Summing up, Applying DDD is a good panoramic on practicing Domain-Driven Design, but also on implementing patterns and using modern tools and approaches like Dependency Injection and Test-Driven Development. It can also be read along or after the original DDD book to improve the understanding of Factories, Repositories and other domain patterns.

English mastering for the average developer

English is the ruling language in the Internet. While many websites are localized for various cultures, the English language is the lingua franca for international business: let's take a look at how it pervades the Information Technology world and why an engineer should master it.

This blog is written in English as it is scoped for the entire world, particularly the American following, despite being managed by an Italian developer like me: I started this project also to master the language and improve my reading and writing capabilities.
However, [International] English predominance is reflected since the origins of computing: the first 128 Ascii codes can represent all the combination of english words and classic Qwerty keyboards are built with this standard 100+ keys set. Today UTF-8 still mantains compatibility with the first part of Ascii, while the encoding of diacritics mark change with the chosen character set.

English keywords are also the base of all the mainstream programming languages: for and while are reserved in every imperative language and Java libraries and C/C++ standards are written in English code. Not only the imperative languages are designed in English, but also declarative ones like html and css: the main tags of this page are head and body, which are common words.
Some technical words have ridiculous or no direct translation in other european languages: the Italian implementazione is mutuated from implementation and I assure you that it sounds very strange. Other neologism are directly imported as-is without trying an unplausible translation: radar and laser are English acronyms.

Many web applications are available in an English version first and are later localized to support user from all over the world. Also a large part of Internet material is written and consumed in English (automatic translations are still not a good deal): free and non-free ebooks are distributed in English. Leaving out the web, English is the lingua france of engineering documents like data sheets and technical specifications: Politecnico di Milano requires a foreign language exam (usually substained in English) before a student's graduation.

Finally, open source communities, which supports projects like Ubuntu, adopt English as the official language, while there are smaller communities and forums localized in Spanish, Chinese, Russian. Documentation is sometimes translated in more than one language, but keeping updated a manual in 10 languages is an hard task and I always use the English version of the php manual. Some documents, like W3C specifications or licenses, have no value in the translated version and must be used in the original English form.
I hope I have convinced you that English mastering is a fundamental skill for a software developer: I live in Italy and I speak Italian as mother tongue, so you can trust I am not Cicero pro domo sua...

10 orm patterns: components of a object-relational mapper

An Orm is a complex and generic tool which stores objects such as Entities and Value Objects (Customer, Groups, Money classes) in a relational database like MySQL or Sql Server, using metadata provided along with the Domain Model. The internals of an Orm usually follow some useful patterns that a developer should know to understand what is going on under the hood and here's a list of the most famous ones.

• Table Data Gateway: an object that represent a table of the database, on a one-to-one basis. It is usually built as a generic class which can be subclasses or instantiated for any physical table.
  
• Active Record: the most common approach, transforms a row of a table in an object. It strictly couples the object structures to the database tables by making the domain object subclassing an abstract implementation.
• Data Mapper: a real Orm is an instance of a Data Mapper, a tool that stores objects which are ignorant of the environment where they will be kept, thus decoupling them from persistence concerns.
  
• Unit Of Work: maintains a list of dirty objects and writes out the changeset. The purpose of this object is to keep track modified data on the entity object that it knows and its flushing capabilitities substitute the save() method of the Active Record implementations. It is a more resilient and sophisticated pattern than Active Record since it strives for persistence ignorance.
• Repository: the persistent-ignorant equivalent of the Table Data Gateway. While a single repository implementation is aware of the database backend, a generic interface is placed between service classes and object retrieval mechanisms to aid decoupling.
  
• Identity Map: a map of objects organized by class and primary key. It is a first level cache that contains every initialized object, so it can be used to prepare a changeset of database queries on flushing.
  
• Lazy Loading: substituting a domain object or collection with a subclass that loads data on the fly when methods are requested, before forwarding the original call.
• Query Object: a class that represents a query for retrieving objects, encapsulating Sql or other high level languages.
  
• Criteria Object: a class that represents a set of criteria for selection of objects of a particular model.
  
• Single Table Inheritance / Class Table Inheritance / Concrete Table Inheritance: patterns implemented to represent class inheritance in relational tables. Favoring composition over inheritance is a must because neither of these patterns does a perfect job in persisting data efficiently and in a clean way.

I hope you'll check out more information on the patterns you are using, maybe without even know that they exist. Feel free to add other useful patterns and approaches to the list in the comments.

Going on vacation

I am going on vacation tomorrow in Romagna, so I will not update Invisible to the eye until August 20.
Stay tuned and subscribe to the feed if you want to be informed of new articles as soon as I return and publish new posts.
I hope you enjoy these holydays!

How to not waste time surfing blogs: introduction to feeds

Have you ever returned to a blog every day or hour to see if the author has written some new, great post? What if the blog you follow are more than an hundred? When too much components are involved, the Publish/Subscribe pattern comes helping, to decouple the subscribers (readers) from following every publisher (blogs) and conserve some productivity instead of surfing an hundred blogs to seek updates.

Publishing
When a website has to syndicate a list of items which is frequently updated, it can place an xml document conforming to certain standards where these items are kept (usually only the last ones published, and with a shortened description in place of the content to keep the document lean). The typical formats used are Atom and Rss, and since the main feed aggregators supports both they are somewhat equivalent.
The list of items can be everything: Invisible to the eye, this blog, publishes a feed for the new posts and for the comments; Wikipedia publishes the list of changes to a page; Twitter the list of an user updates.
Feeds are normally generated by a server-side script, in my case the same that manages blog posts; it can derive feed fields (title, text, url) from the posts table in the database. When a feed is in place, the tedious process of open the main page to check if new articles have been published is standardized and can be automatically done by a program or a web application for us.
The last step is to take advantage of autodiscovery: placing a special link tag in the head section of html documents can link to a xml feed which will be displayed by browsers as available for subscribing. In Firefox, for instance, a small icon appears in the location bar where a feed exists as an alternate version of a page content.

Subscribing
On the other side, feed aggregators manage a list of feeds you subscribed to. When you discover a feed, you can put the url in a program or web application that will check periodically, every few hours, the feed for you.
Every time the aggregator gathers data from feeds, it lists the new items that you have not already read and organizes them as you like. When you want to check your list of 100 preferred blogs for new content, the only action needed on your side is to open Google Reader and see the list of new articles. Every post has a globally unique url which act as a primary key.
I prefer web application for aggregating data since they mantain the list of current unread items in every machine where I'm viewing them. Ideally, they can also do only one ping for every feed even if thousands of users are subscribed: they are the Blackboard of this pattern.
In a real publish/subscribe system the websites should notify the aggregator of new items, but the http technology is limited and the inverse process happens to emulate a push style. However, some extension to this mechanism is in development.

Wrapping and mashups
Once a standardized way to let content flow such as xml feeds has been developed, new ideas come up and xml is a pillar of web 2.0:

• FeedBurner is a wrapping service that envelopes a feed giving you a nice, short url: then you can link in your pages the FeedBurner version and gain automatic statistics and optional reformatting, and also embellishment of the syndicated data when viewed in-browser (with links and buttons to pass the feed to the most famous web based aggregators).
  
• A mashup is an application that combines external sources of content and provides a unique view of them. For instance if you are a fan of php, you can subscribe to a feed of a php mashup which publishes the best php articles it finds scanning thousands of feeds around the web, filtering out the bad choices.
  

Now you have a general view of how xml feeds works. I hope you will join the web feed of this blog if you want to remain informed of brand new articles.
And if you do not have a feed aggregator yet, check out Google Reader.

Twitter vs. Facebook: how to manage a downtime

Yesterday's DDoS attacks on Facebook and Twitter exposed the critical addiction to social networking and the fragility of high-loaded web applications. It is not clear whether there is the same people behind the two attacks, but the system managers and their infrastructures responded differently to the issues in outage, and gave a diverse impression to the end user.


According to Wikipedia, a DDoS is:

A distributed denial of service attack (DDoS) occurs when multiple systems flood the bandwidth or resources of a targeted system, usually one or more web servers.

and that's what happened on August 6. Often the "multiple systems" are remotely controlled by a trojan horse or some similar malware, and are unaware they're being maliciously used.
Let's explore some differences on how both systems reacted to the attack:

• Facebook uptime was less affected than Twitter one. Although it depends on the particular user point of view, it seems from reports that Facebook shows slowdowns and latency increase, but less significant pages that fails to load in respect to Twitter total outage Personally, I encountered some error pages when using a pair of application, but no unreachable main pages (profile, groups) in the period of time when I used Facebook yesterday (afternoon and evening), and it has to managed the users that turned to it while being unable to tweet.
  
• Facebook and Twitter are built on different stacks, althoug it can be a coincidence. Facebook is all php, while Twitter is built on Ruby on Rails. The effective difference depends on what resources the attackers saturated: bandwidth is not dependent by the technology you're using on a webserver, while cpu load and memory usage are affected.
  
• Facebook shows some errors, while Twitter not. As I said, I encountered some error pages while using Facebook while Twitter was failing and letting the http request time out. This was particularly fastidious because of widgets that performs a twitter mashup: I was forced to remove the Last tweets column from the sidebar of this blog because it would never load, expanding the downtime of Twitter to Invisible to the eye.

The last point is a tip on how to annoy your users. Facebook had a 99.9% uptime in 2008, while Twitter has a tradition of being unrechable every once in a while. The statistics shows that in the last year the availability of the service was barely ~99%, with 84 hours offline (despite improving from the six days of outage in 2007). Since the uptime is measured in nines, it is much, and since it seems that Twitter might not know when it is offline, another website has been created to monitor it.
Facebook probably has better servers, or more scalable technology: when even the fail whale is unreachable, Facebook will be still here.

Doctrine 2 now has lazy loading

Lazy loading is the capability of performing a expensive operation on demand, only when it reveals necessary from a client request: in the Orm field, the expensive operation is the loading of an object graph part. In Doctrine 2 I made some architectural choices implementing the proxy approach, which substitute a subclassing object to every association which is not eager-loaded.

Disclaimer: there is not a stable Api for Doctrine 2. This is a design post about how I coded the lazy-loading features with the help of the lead developer Roman Borschel. The name of methods or classes can slightly change in the future.

*-to-one associations: dinamic proxies
The technical solution for a one-to-one or many-to-one association is to use a dynamic proxy, an object whose class is generated on the fly, subclassing the original one. I talked about this approach extensively in the previous post about lazy loading. However, I feel to make some precisations developed while applying in practice the theoretical approach discusses there:

• the example was about subclassing a Group class to Group_SomeOrmToolNameProxy, and inject a proxy object as the User property ($user->group). This class is generated only the first time the lazy loading capabilities is used with Group as a target entity, and it is saved in a temporary folder to be recycled in subsequent requests. This folder should be cleaned out when rolling out new code.
• if the dinamic proxy class is already present, there is no need to generate another one and an attempt to redefine it would raise a php fatal error.
• the proxy object should contain foreign keys of the source object ($user in the example), put there when fetched. The original entity class does not contain fields to store foreign keys as it is persistence-agnostic, so the more cohesive class to place them is the proxy one.
• I do not enter in the detail on how the proxy loads itself, but a reference of an Doctrine\ORM\Mapping\AssociationMapping subclass is passed to it in the constructor. This allows independent unit testing of the proxy behavior and of the effective hydration of data in a object (load() method on AssociationMapping).
  

*-to-many associations and the need for a Collection interface
While generating a proxy for a one-to-one or many-to-one association is mandatory to fulfill the same contract of a complete graph, is somewhat simple to satisfy the loading of collections of objects. In Doctrine 2, entities are required to implement collections in their fields with an instance of Doctrine\Common\Collections\Collection interface, and this is commonly done with instancing in the constructor Collections\ArrayCollection; when reconstituting an object from the database, a Doctrine\ORM\PersistentCollection instance is substituted in hydration and it has the mandatory field reference to the AssociationMapping object and the EntityManager to load itself when required to do so.
The Collection interface is not orm-dependent and it has been placed in the Common namespace to let the user build a real persistence-ignorant Domain Model.
A quick solution to implement lazy loading would be to fill the PersistentCollection instance with dynamic proxies. However, this proves to be slow as every object in the collection will issue a different query to the database for retrieving its internal data, and requires to join association tables even if the collection is not used.
Instead, the current implementation injects in the PersistentCollection (which obviously implements Collection) the needed collaborators, as no constructor is specified in the interface:

• the collaborators are always EntityManager and the AssociationMapping instance.
• there's no need to store foreign keys in the PersistentCollection since *-to-many relations do not use foreign keys from the source object. In our example, the Groups a User belongs to are joined with the primary key of User, and other collections will do the same.
• also here the unit testing of the PersistenceCollection trigger capabilities is separated from the loading itself, performed by an AssociationMapping instance. There are also functional tests to run the overall process of lazy loading in its entirety.

Final notes
Remember that lazy loading is a handy feature, but can be easily abused. Performance can suffer when the load is performed as many queries are issued as needed instead of few, eager queries which hydrates the part of graph you need to work on: this is the point of join() in the Doctrine\ORM\Query class. Enabling lazy loading probably will make your php script more chatty but save time when not all the objects are needed.
Doing an eager load can be impossible if the relations are bidirectional, like in the User-Group example: try to simplify your model removing one side of associations when not strictly needed.
I hope you will enjoy using Doctrine 2 and its lazy-loading feature and I think I've done good job of implementing it and explain the architectural issues I've encountered. Feel free to ask any question I have missed out.

What drives development nowadays

Many blog posts (also of mine) contain acronym and abbreviations which have spreaded in the agile world. Every now and then words like TDD, BDD enter feed readers which might have still to learn what they mean.
To help fellow readers and to make a bit of humour on the number of methodologies which are growing, I made a research on all the techniques that shares the last part of the name.


Here is the list:

• TDD - Test-driven Development is a development technique based on prewritten test case and short coding iterations (5 minutes). The mantra of TDD is red-green-refactor, where red means that a test fails and green that it passes.
• BDD - Behavior Driven Development is a response to TDD which involves the definition of scenarios and tests involving different business contexts and units. The mantra of BDD is given-when-then.
• FDD - Feature Driven Development is the typical agile approach: develop some features at the time, in incremental steps. An iteration is the basic unit of time.
• PDD - Process Driven Development is a methodology for constructing non-trivial custom software applications by analyzing the business processes in which they will be used.
• DDD - Domain-Driven Design merits a mention although it is not a type of Development. The focus of this methodology is to not worry about the infrastructure layer which comprehends persistence, authenthication, etc. but to develop a powerful Domain Model which has no dependency on frameworks or libraries.

There are also less serious approaches, which are anti-pattern:

• ADD - Asshole Driven Development: a methodology where the biggest jerk makes all the big decisions.
• RDD - Resume Driven Development: a situation where programmers wanted to use tools, techniques, or technology to improve their resume instead of solving the customer’s problems.
• FDD - Fear Driven Development: there are different definitions of this anti-pattern, but I prefer when you have an amount of LegacyCode and maintaining the code is hard because you don't know what unintended effects your changes have.

This is a summary of all driven development approaches which I have encountered around the Internet. Do you know any other?

Coding an interface

Interfaces are a contract that a class must respect: a set of methods with specific signatures that should be implemented. Many components become dependent on interfaces that abstract away the concrete classes: mistakes have to be maintained and inherited.

In object oriented languages, an interface I corresponds to a set of methods (without body) which acts as a specification for classes. A class C that implements I must provide a body for every method that I has. The objects of class C are not only instanceof C but also instanceof I.
Interfaces are present in languages such as Java and Php. Since there's no limit to the number of interfaces that a class can implement, multiple inheritance is obtained via interfaces.

Here is a checklist for writing a clean interface:

1. Keep methods number to a minimum. The complementary of an interface is an implementation, and every implementation is forced to provide code for the all the methods specified. If you are extracting an interface from a particular class, consider to only a cohesive set of public methods, which lead us to the next point. E.g. a Queue interface should have only two methods: put(...) and get(...). Specifiy a constructor or a length() method is not very purposeful.
   
2. Segregate interfaces. The interface segregation principle says that a class should not depend on interfaces that it does not use. The meaning is that when an interface contains many methods, probably there are two or more interfaces which it should be broken into. E.g. Php SplQueue and SplStack could be interfaces instead of concrete classes, with only the purposeful methods included and SplDoubleLinkedList implementing both. As they are now they expose all the Iterator methods, which have no sense on a real stack or queue.
   
3. No static methods. Interfaces provides a contract for polymorphism, and static methods are only global procedures under a nice name. Althoug some languages like php don't complain when encountering an interface definition which contains static methods, they should be avoided. E.g. resist the urge to put a static count() method on Queue to count the instantiated objects. Create a AbstractQueueFactory instead with a count() instance method.
   
4. Abstract the meaning. Particularly if you are extracting an interface, pay attention to the method names: they have to carry an abstracted meaning and not implementation details, so some of them needs to be renamed. E.g. the Spl Iterator interface has generic methods like current() and key(); it does not assume anything on the keys of the elements which the iteration is performed on: in subclasses they can be filenames, array offset, object property names and so on.
   
5. Do not tie to concrete types. Depending on an interface decouple a system from the implementation of a particular service, but if the interface depends on other concrete classes the advantage is lost; the only classes cited in an interface definition should be Value Objects (like java String or php Datetime) or Entities (the example User class, without injected services). E.g. an Authenticator interface with a method signature which contains a Zend_Db object is useless.
   

I hope you'll be writing small and cohesive interfaces now. Let me know what are your guidelines for coding interfaces in the comments.

5 books to change your point of view

For a software developer that face the challenge of creating maintainable and extensible applications, which provide value to users, there is the need to learn something every day to keep up with innovation. Programming books are a resource to explore in depth a subject.
Books which treat a topic at a great extent are the simple tool to teach yourself practices, skills and technologies; but there are also classics (or wannabe in the future): each of these books paradigm-shifted my mind and I hope you can find in them some of the answers you're looking for, as they challenge the naive assumptions of software development.

1. The Pragmatic Programmer: From Journeyman to Master: this is one of the most practical books that I've ever read. It is free from mumbo-jumbo buzzwords and focus on a pragmatic approach to software development, a compromise between purity of the design and getting things done. It also explains Extreme Programming/Agile practices and is a complete read despite being only 300+ pages long; it does not cover in full depth its topics but it goes as far as possible to tackle the advantages of source control, refactoring, testing, plain text, and more. It was the first book for me that gave a sense to the agile word.
2. The Mythical Man-Month: Essays on Software Engineering: a classic from Fred Brooks. The man-month is the unit of measure for the work needed in a project: Brooks was the first to say that adding manpower to a software project would slow down the process. Even the Halloween documents commentaries cite The Mythical Man-Month; Plan to throw one away comes from here.
   
3. Patterns of Enterprise Application Architecture by Martin Fowler. While the original GOF Design patterns book it's about common repeatable object structures in everyday applications, Fowler's one talks of the infrastructure layer: ActiveRecord, DataMapper, Unit Of Work and so on, fighting the object-relational mismatch, presenting distribution and layering. After reading this book you will feel a compelling desire to write an Orm, and you will be more or less capable of doing it.
4. xUnit Test Patterns: Refactoring Test Code is a writing in all the patterns that a developer encounters when writing unit tests with an xUnit tool (phpunit, junit, nunit...). Who watches the watchmen? applies also to the test vs. production code duality and the maintainability of a project test suite is crucial for having it run by developers. The term fixture started to have a sense when reading this book.
   
5. Dive into Python, an introduction to the Python language, the one that Google uses. This is a free ebook and explains the basics of this duck typing total-object-oriented language, where 2 and True are objects and magic methods are born.
   

Maybe you have already a copy of some of these next to (or in) your development box.
What books made a great impression on you? I'd like to read what changed your way of thinking. Let me know in the comments.