How do I organize my GIS projects?

from http://gis.stackexchange.com/questions/18199/how-do-i-organize-my-gis-projects

Base Files/DBS

• These are files that are "raw" in nature and constitute the base of all my analysis
• These files, databases, and data are hosted outside of my projects folder, and are hosted on my internet server, local computer, and dropbox. I always have access to them, and they are very organized, dis and aggregated. You'll spend a lot of time organizing these.
• I put them all in databases whether in Arc or PostGIS.
• To each table, I add 3 fields in the table itself or the meta data: DATE_OBTAINED, DATA_DATE, SOURCE_NOTES
• Also Base files could be queries of multiple other tables. For example, a table could aggregate all the traffic count I obtain into one large query/table.
• I also put here all other data that I find scouring the internet.
• I NEVER DO ANY DIRECT ANALYSIS ON ANY OF THE FILES IN THE BASE FILES

Project Files

• All my project files go in a my_projects folder. It contains everything related to that project as in, if I copy and paste that folder somewhere else, it will contain everything.
• Usually I have the following structure:
  • my_project/
    • admin/
    • communication/
    • raw_data/
    • analyzed_data/
    • output_data/
    • from_client/
    • FINAL/
    • code/
    • some_document_date_time.doc
    • README
• Slowly I've been moving to a local GIT. (you can even host it locally or on your own server). The reason I do not put it on GitHub is that github has a 1.2gb limit which is useless for GIS analysis
• For my projects, I usually replicate all the GIS tables that I need for my analysis into a new db: project_whatever.
• 9 times out of 10, I work only in shp files and I save all my GIS (images, excel, coordinates, etc) to my projects/my_project/raw_data, projects/my_projects/analyzed_data, andprojects/my_projects/output_data.
• When a project is complete, I put the final submitted copy inmy_projects/FINAL/date_submitted
• For my MXD, I usually save to a new MXD every 2 or 3 hours my_proj_dec_22_11__13_20.mxdfor example
• For Ms Word documents, illustrations, and mostly editing documents, they go in my_projects folder such as RFP_TENDER_Dec_22_11__11_15.doc and draft_ver5_Dec_31_11__12_30.doc. Again all my final deliverables go in in the FINAL folder
• For R, Python code and some C#, it gets a bit tricky, as I host it outside of the project but with a working copy to my_projects/code folder. I do this as most of the python code is reusable. If you put all your python code besides the projects, you'll forget about them. Also, all my python code goes on github.
• To me project files include any file types including time tracking, communications (I save all my emails as .msg files), I log all our verbal communications in a word file, and I put all those files my_project/communication
• With ArcGIS use Models, LYR, and "save selection as a new shp layer". These tools will make it easy to store files in smaller formats, reuse files, and with models, be able to use something in another place.

Final Output

• Each project when finalized gets zipped and put on my external harddrive.
• All final products are converted to files from Tables, and to PDF from all other formats.
• Every Project I do, gets printed for a hard copy backup

The Bottom Line

• Each person uses multiple and different software and tools. A lot of people I know get organized using basecamp, Harvest, or any other multitude of tools. Also people have different working habits and OCD tendencies. I'm fairly obsessed with getting stuff organized maybe a bit more than others. So develop the system that causes you the least stress while guaranteeing you'll be consistent in applying and updating it
• Backup and replicate everything
• Don't work directly on your raw / base data
• For your projects always use a replica file, as data changes over time, and you don't want to be scrambling to find the base_layer_2006.shp.
• each my_projects folder must have a README text file that you edit it while you're doing the projects to give some basic information that you know you'll forget later when you visit the project 2 years down the road