Optimal Energetics – My PhD

Performance Simulation / Energy Systems / Programming / Optimization

Python for Energy Modeling Part 3 – Post Processing DRAFT

Posted by BenjaminMarcusJones on January 16, 2012

Welcome to the next installment of of Python for Energy Modeling!

Last time we looked at scripting the creation of many input files for a parametric study of a building energy concept. Now we will turn to the post-processing side of the process. Often, a spreadsheet tool like Excel is an energy modeler’s first choice for many analysis tasks. This is great for small cases, but if the number of files or the amount of data is large, Excel will cost you time and lead to errors. This is where you should turn to Python, or a similar scripting language!

Let’s look a concrete example, from a project I worked on a few months ago. What we needed to do was generate building load profiles for 3 stock building geometries. We were investigating 8 different internal loads (office, residential, etc.), and various other parameters including wall insulation. After an input file generation script similar to last week’s, I had 384 TRNSYS input files! These were executed over night resulting in 384 output files. One of these output files is below;

 TIME                    	SQHEAT                   	SQCOOL                   	
  +0.0000000000000000E+00	  +0.0000000000000000E+00	  +0.0000000000000000E+00	
	  	  	  ... MORE ... 
  +5.7000000000000000E+01	  +1.3237372413443035E+05	  +0.0000000000000000E+00	
  +5.8000000000000000E+01	  +7.7162712393718830E+04	  +0.0000000000000000E+00	
  +5.9000000000000000E+01	  +6.5336426881896477E+04	  +0.0000000000000000E+00	
  +6.0000000000000000E+01	  +6.7508927183426335E+04	  +3.5916664027267243E+03	
  +6.1000000000000000E+01	  +6.3983667187962485E+04	  +4.5933382762949959E+03	
  +6.2000000000000000E+01	  +4.7441687548894530E+04	  +1.5935811219563341E+03	
  +6.3000000000000000E+01	  +3.3644472803687437E+04	  +2.9517575364511831E+03	
  +6.4000000000000000E+01	  +4.2939548284044671E+04	  +0.0000000000000000E+00	
  +6.5000000000000000E+01	  +5.1990203122431951E+04	  +0.0000000000000000E+00	
  +6.6000000000000000E+01	  +5.9902003142972004E+04	  +0.0000000000000000E+00	
	  	  	  ... MORE ... 
  +8.7600000000000000E+03	  +1.8288512650404076E+05	  +0.0000000000000000E+00	

This file is a typical output file from TRNSYS, a tab-seperated ASCII text file with a single header line. What we have in each row is the time stamp (8670 hours of the year), and then the average heating and cooling load for that hour (kJ/hr).

From these 384 output files, we might ask; what building instance requires the most heating or cooling over the year? What is the correlation between average envelope U-Value and the system loads? Generally, our problem statement is; How do we statistically analyze and summarize a large number of output files?

Once again, let’s write out some pseudo-code;

# Specify directory and file extension

# Collect the file names of interest from the directory

# For each file found, process each in turn
    # Open it
    # Read it as a comma separated style format
    # ! Skip the header line !
    # For each row of the file
        # Access the 2nd and 3rd column
        # Convert [kJ/hr] into [Watts]
        # Save the data to our list
        # Calculate the cooling and heating load in [Wh]
        # Calculate the average load in [W]
    # Close the file
    # Write results for the processed file

We have a similar block structure as last time; a problem parameter section (line 1 and 3), and then a looping structure. This looping is exactly where Python does the repetitive boring tasks to save you your sanity!

Looking a bit closer at lines 4 – 16, we can see that there are two loops going: first, we loop over each file that was discovered in our project directory. In this case, 384 ASCII text files with the extension “.out”. Note also that the “.out” files have one line of header data, this header will be skipped. The file is further scanned and the data is extracted. Once these data are saved, we can perform any conversion and statistics we need.

So let’s start scripting!

import os
import csv

# Specify directory and file extension
searchDirectory = os.path.normpath(r"C:\\Project5\\Output")
searchEnding = "out"

First, I am importing some “helper” functions to make this script easier. Specifically, I am using the “Operating System” module, which makes it easy to access both Windows and Linux paths.

Note also the “\\” double back-slash, and the r”” string. In Python, the “\” character is a special escape character, used to represent other ASCII characters like tab (\t), carriage return (\n), nil (\0), etc. The r in r”string” tells python that this string doesn’t have any escape characters. So I could also use single slash. But as a reminder, I make it a habit of using “\\” regardless. Finally, I am using the os.normpath() function to make 100% sure my path is in a proper format. This is triple redundant, I do this because I have too often made the mistake of improper path names!

Finally, we want to ignore anything in the search directory that doesn’t have the “.out” extension”.

Next, I want to scan this directory and store any “.out” file paths;

# Collect the file names of interest from the directory
inputFilePaths = list()
for filename in os.listdir(searchDirectory): 
    if filename.endswith(searchEnding):
        # Found a file with proper extension
        fullFilePath = os.path.join(searchDirectory, filename)
        # Add it to our running list
        inputFilePaths.append(fullFilePath)

# Some feedback on what was found
print "Found {0} '{1}' files in {2}. ".format(len(inputFilePaths), searchEnding, searchDirectory)

First, I create a blank list in order to store all the paths I find. Try this code without first declaring your list! Python is normally very forgiving about declaring and changing variables later in your code (this is called “dynamic typing”, and is one reason why Python is so flexible and “easy” to program), but in this case we are adding subsequent elements to a list. Matlab programmers will find this behavior familiar, and the discussion of “pre-allocating” arrays arises.

We take advantage of the os.listdir() method, which, yes indeed, lists files in a directory. From this list of filenames found, we loop over each one and check if it endswith() our extension “.out”. If we do find a match, we store this full path in our list. Since listdir() only gives us the file name and not the parent directory, we need to join() them together.

Finally, let’s provide some feedback for this section (line 17) and show how many files were found. Note the syntax of the print “{}”.format() statement. This is an alternative way to print and format strings with variables included, and is preferred for flexibility and control. You can also just list strings and variables together in a print statement; but this can get awkward…

# For each file found, process each in turn
for filePath in inputFilePaths:
    # Open it
    openedFile = open(filePath)
    # Read it as a something separated style format (\t is TAB)
    openedCSV = csv.reader(openedFile, delimiter="\t")
    # ! Skip the header line !
    openedCSV.next()
    # Blank list for storing data
    coolingLoad = list()
    # For each row of the file
    for row in openedCSV:
        # For each row, access the 2nd column
        # Convert [kJ/hr] into [Watts]
        thisHoursCoolingLoad = float(row[2]) / 3.6
        # Save the data to our list
        coolingLoad.append(thisHoursCoolingLoad)
        # Calculate the cooling and heating load in [Wh]
    totalCoolingLoad = sum(coolingLoad)
    # Close the file
    openedFile.close()
    print totalCoolingLoad 

Here we can see this same looping structure, first over each file, and then over each line of each file. The first part is to open the file (line 21) and use the “csv” module to provide some additionaly functionality – the capability to read in a file line by line, with a certain delimiter (TAB in our case), and to return a list of items found.

Next, the file is scanned row by row, returning the list of items. In our case, we have 3 items; TIME, SQHEAT, and SQCOOL. For the sake of brevity, I am only accessing the cooling load in this example. This is indexed in the list by [2]. Python indexing starts at 0, therefore [2] is the third list element! Also in line 32, I first convert the “+1.32+05″ style strings into python floating point, and then converting this number from [kJ/hr], which is a native TRNSYS unit, into more comfortable [W]. This number is then stored to a running tally of cooling loads (in other words, the entire 3rd column of the data table.

Finally, for each file, the column list is summed, and printed.

Bonus Excercises!

  1. How would you modify this script to save the summary results into a new file?
  2. Another interesting question is – at what time is the peak cooling or heating load?
  3. Another helper module that I often use is the NumPy , one of the most exciting modules for us as energy modelers or engineers. This module, and others matplotlib, SciPy), promise a free open source alternative to i.e. Matlab, but with the full power of Python as a bonus. What elements of NumPy would make this script more general, clear, or useful?

Now we have looked at both sides of the energy modeling work flow, pre-processing and post-processing. Hopefully, you can see how powerful this combination of Python+Simulation is! Next time, we will review the overall process of scripting, best practices, and issues like compatibility and installation!

Cheers from Vienna,

Marcus

Posted in Programming, Uncategorized | Tagged: , | Leave a Comment »

Python EC2 update p3

Posted by BenjaminMarcusJones on January 7, 2012

Continuing with the Boto package for accessing Amazon cloud with python. I was researching more regarding the python logging capacity, which is an indispensable module for all of my code. The issue was that the Boto package also supports logging, and dumps the (I think RPC) communication logs onto the screen, making it impossible to read my own log messages. The workaround so far is to set logging to INFO level, avoiding the Boto DEBUG log messages. Ideally, I would like to log Boto messages to a file, and keep my messages on the console. But time to move on!

To pick up from last entry, we had successfully started basic communication with Amazon S3 and EC2. Now to start some instances and get them talking.

I’m now following the book “Python and AWS Cookbook“. Here I found a helper function “launch_instance”, which launches a specified AMI on EC2. You can checkout all the source from the book through a GitHub repository!

  • Created ec2_launch_instance module with verbatim code from the book. here’s the method signature and all it’s default values for reference;
    def launch_instance(ami='ami-7341831a',
       instance_type='t1.micro',
       key_name='paws',
       key_extension='.pem',
       key_dir='~/.ssh',
       group_name='paws',
       ssh_port=22,
       cidr='0.0.0.0/0',
       tag='paws',
       user_data=None,
       cmd_shell=True,
       login_user='ec2-user',
       ssh_passwd=None)
    
  • Ran into a dependency from boto.manage.cmdshell.
    ImportError: No module named paramiko
    

    Tracking this down on PyPi; “This is a library for making SSH2 connections (client or server).” Easy_Install fails, I think it’s a sub-dependency PyCrypto. This causes a further error;

    error: Setup script exited with error: Unable to find vcvarsall.bat
    

    Which means a C compiler is needed. Following some instructions (and I’ve seen this error before) I installed MinGW, and set a distutils.cfg file. Now there is a new error:

    RuntimeError: chmod error]
    

    Which is asking for a linux system. So this path isn’t going to work… Welcome to Python!

  • Always easiest if someone else has created windows install binaries, so we are in luck with the kind gentleman at http://www.voidspace.org.uk! Downloaded and installed for 32 bit, python 2.7. Maybe someday I’ll figure out my nemesis, the MinGW compiler… For now, let’s try Paramiko again;
    easy_install paramiko
    

    Works.

  • Need to pass into the function my specific security credentials (logging into online console again);
    launch_instance(key_name="PrivateKey",key_extension='.pem',key_dir="C:\EC2",group_name="default")
    

    And many other permutations all fail! The instance does start up (I probably have around 10 running now…), but the SSH client is not properly accessing the RSA private key file.

  • Now I have the problem of likely having one or more instance started, need to close them all. Fortunately the cookbook has just the function;
    find_all_running_instances
    

    Just as I suspected: I have 8 running instances! This is where EC2 can get expensive – if you forget these for a few months… Once again, the online console is no help at all here, it shows no instances.

  • Another way to list instances;
    ec2 = boto.connect_ec2()
    print ec2.get_all_instances()
    

    Which returns

    [Reservation:r-61521500, Reservation:r-ab5e19ca,...] 
    

    for all 8 instances. These are Reservation objects.

  • Now I’m really curious about the nuts and bolts behind all of this. I hacked together a small script to inspect the objects here, first a single Reservation object, where I learned that a Reservation has an “instances” property. Then I learned the “instances” is a list of Instance objects, and in my case the list is len=1.
    ec2 = boto.connect_ec2()
    allReservations = ec2.get_all_instances()
    
    print (allReservations[0])
    print type(allReservations[0])
    print dir(allReservations[0])
    print ""
    print allReservations[0].instances[0]
    print type(allReservations[0].instances[0])
    print dir(allReservations[0].instances[0])
    

    Which results in (abbreviated listing);

    Reservation:r-61521500
    <class 'boto.ec2.instance.Reservation'>
    ['__class__', ..., 'id', 'instances', 'item', 'owner_id', 'region', 'startElement', 'stop_all']
    
    Instance:i-d4661db6
    <class 'boto.ec2.instance.Instance'>
    ['__class__', ..., 'id', 'image_id', 'instanceState', ..., 'stop', 'terminate', ...]
    
  • So there are a plenty of methods and attributes. Particularly interesting, as the clock keeps ticking on my running instances, is Reservation.stop_all(). Therefore, here’s a way to stop all running instances;

    ec2 = boto.connect_ec2()
    allReservations = ec2.get_all_instances()
    for reservation in allReservations:
        reservation.stop_all()
    
  • On another note, I’m installing ElasticFox for Firefox 9 (the original is not supported on Firefox 9).
  • Posted in Parallel, Programming, Uncategorized | Tagged: , , , | Leave a Comment »

    Python for Energy Modelers – Part 2 – Simple Pre-processing

    Posted by BenjaminMarcusJones on January 7, 2012

    I`ve been asked to guest write a series on Python and Programming, here is an excerpt! For the full post, head over to openrevit!

    I am going to start by looking at a common task in an ASHRAE 90.1 baseline model (for LEED or similar rating system); creating four input files for each of the four rotations. I do these models on a weekly basis, and this was one of the first things I wanted to automate.

    Let’s first identify the problem in one sentence: We want to create four input files with four different rotation angles; 0, 90, 180, and 270, based off of a common “template” file. We are going to use EnergyPlus here, but this method works for any text based simulation file where you can access the angle parameter!

    Here is our “template” file section of interest:

    !-   ===========  ALL OBJECTS IN CLASS: BUILDING ===========
    
    Building,
        Baseline 0,              !- Name
        $$ReplaceThis$$,         !- North Axis {deg}
        CITY,                    !- Terrain
        0.040000000000000001,    !- Loads Convergence Tolerance Value
        0.40000000000000002,     !- Temperature Convergence Tolerance Value {deltaC}
        FullExterior,            !- Solar Distribution
        25,                      !- Maximum Number of Warmup Days
        ;                        !- Minimum Number of Warmup Days
    

    Notice line 5, where I have edited the EnergyPlus rotation angle field and used my own very unique text value; “$$ReplaceThis$$”. This is a common and simple technique in parametrization of a text input file.

    Full details continue here!

    Posted in Tutorial | Tagged: , , | Leave a Comment »

    Python+EC2 update p2

    Posted by BenjaminMarcusJones on January 3, 2012

    After successfully working from the Java based command line EC2 toolset from Amazon, we need to start automating instance management with Python. We’ll be using another tutorial for the “boto” python EC2 wrapper module.

    1. First, install the amazon web services library python wrapper;
      easy_install boto
    2. Created boto.cfg;
      [Credentials]
      aws_access_key_id = {ACCESS KEY ID}
      aws_secret_access_key = {SECRET ACCESS KEY}boto
      

      Saved it to the c:\EC2 directory.

    3. Created a windows environment variable to point to the .cfg file.
      BOTO_CONFIG=c:\EC2\boto.cfg
      
    4. Try a “hello world”.
      from boto.s3.connection import S3Connection
      connection = S3Connection()
      ERROR: boto.exception.NoAuthHandlerFound: No handler was ready to authenticate. 1 handlers were checked. ['HmacAuthV1Handler'] Check your credentials
      

      This is a problem with the windows environment variable and the boto.cfg. Tried many fixes; using AWS_CREDENTIAL_FILE variable from StackOverflow, typing SET VARIABLE, etc. Nothing works.

    5. Alternatively, can pass in the access keys directly;
      from boto.s3.connection import S3Connection
      connection = S3Connection('XXXXXXXXXXXXXXXXXX', 'XXXXXX/XXXXXXXXX/XXXXXXXXXXX')
      

      Works!
      A few more checks:

      Started Boto EC2 test script
      S3 Buckets: []
      S3 Regions: [Zone:us-east-1a, Zone:us-east-1c, Zone:us-east-1d]
      Finished
      

      Note that we are back in us-east by default.

    6. After some more digging, was able to successfully load the boto.cfg file. Created a c:\etc\ directory, and put the boto.cfg file here. This is much cleaner for the code and seems to be the preferred way.
    7. UPDATE JULY: Authentication file settings SOLVED
      - In Eclipse, set the run environment with required environment variables:
      Run menu -> Run Configurations -> Environment tab ->  SET BOTO_CONFIG = c:\etc\boto_config.cfg (Or any file path)
    8. Wasn’t able to set the default region in the .cfg file:
      Started Boto EC2 test script
      S3 buckets: []
      EC2 zones from default region: [Zone:us-east-1a, Zone:us-east-1c, Zone:us-east-1d]
      EC2 zones from eu-west-1 region: [Zone:eu-west-1a, Zone:eu-west-1b, Zone:eu-west-1c]
      All regions: [RegionInfo:eu-west-1, RegionInfo:sa-east-1, RegionInfo:us-east-1, RegionInfo:ap-northeast-1, ...]
      Finished
      

      Just something to watch for.

    9. Enable logging;
      boto.set_stream_logger()
      

      Goal is to have the HTML communication stream logged to file, while I log my own code to the console. Not able to figure this out yet with the varying loggers/handlers in python logging module.

    Enough for today! 2 hours.

    Another tutorial; Backup to AWS EBS via Rsync and Boto.

    Another tutorial; Boto Mturk Tutorial

    Posted in Parallel, Programming, Uncategorized | Tagged: , , , | Leave a Comment »

    Python+EC2 update p1

    Posted by BenjaminMarcusJones on January 2, 2012

    This post describes the first step in Programming challenge: Python+EC2. Listed below are the instance pricing and characteristics, followed by a step by step of tutorial progress in starting a fist instance in the cloud.

    Amazon EC2 instance pricing

    Instance pricing on the European (Ireland) server euros per hour. Interesting are highlighted.

    Linux/UNIX Usage Windows Usage
    Standard On-Demand Instances
    Small (Default) $0.095 per hour $0.12 per hour
    Large $0.38 per hour $0.48 per hour
    Extra Large $0.76 per hour $0.96 per hour
    Micro On-Demand Instances
    Micro $0.025 per hour $0.035 per hour
    Hi-Memory On-Demand Instances
    Extra Large $0.57 per hour $0.62 per hour
    Double Extra Large $1.14 per hour $1.24 per hour
    Quadruple Extra Large $2.28 per hour $2.48 per hour
    Hi-CPU On-Demand Instances
    Medium $0.19 per hour $0.29 per hour
    Extra Large $0.76 per hour $1.16 per hour

    Amazon EC2 instance characteristics

    Standard Instances

    • Small Instance (Default) 1.7 GB of memory, 1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit), 160 GB of local instance storage, 32-bit platform
    • Large Instance 7.5 GB of memory, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 850 GB of local instance storage, 64-bit platform
    • Extra Large Instance 15 GB of memory, 8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

    Micro Instances

    • Micro Instance 613 MB of memory, up to 2 ECUs (for short periodic bursts), EBS storage only, 32-bit or 64-bit platform

    High-CPU Instances

    • High-CPU Medium Instance 1.7 GB of memory, 5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each), 350 GB of local instance storage, 32-bit platform
    • High-CPU Extra Large Instance 7 GB of memory, 20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each), 1690 GB of local instance storage, 64-bit platform

    Getting started, running first instance from the command line.

    Following this tutorial to the letter:

    1. Downloaded the EC2 Command Line Tools (Java based command line interface), extracted to c:\EC2
    2. Created and downloaded private key and 509 certificate
    3. Created this batch file
    4. @echo off
      set EC2_HOME=c:\ec2
      set PATH=%PATH%;%EC2_HOME%\bin
      set EC2_PRIVATE_KEY=c:\ec2\PrivateKey.pem
      set EC2_CERT=c:\ec2\509Certificate.pem
      set JAVA_HOME=C:\Program Files\Java\jre6
      "%JAVA_HOME%\bin\java" -version
      @echo on
      
    5. Tested with;
    6. ec2-describe-images -x all

      Works.

    7. Added a key pair to access server environment;
    8. ec2-add-keypair mj-keypair

      Copied to mj-keypair file, added a carriage return after —–END RSA PRIVATE KEY—–, to work with putty.

    9. Starting an instance. Choose the ami-973b06e3 basic 32 bit linux.
    10. ec2-run-instances ami-973b06e3 -k mj-keypair

      Fails.
      My region is eu-west-1, and the default is us-east-1?

    11. Change region
    12. set EC2_URL=https://ec2.eu-west-1.amazonaws.com

      Success (tested with

      ec2-describe-availability-zones

      )

      But still can’t start instance.

    13. Listed instances again in new region, found;
    14. ami-23b6534a    ec2-public-images/fedora-core4-apache.manifest.xml

      And executed

      ec2-run-instances ami-23b6534a -k mj-keypair
      

      Success!

      RESERVATION     r-45127e24      189613975676    default
      INSTANCE        i-9054cdf2      ami-23b6534a                    pending mj-keypair
      
    15. Opening windows firewall ports for SSH and HTTP.
    16. ec2-authorize default -p 22
      ec2-authorize default -p 80
      
    17. Run PuttyGen. Import mj-keypair key. File-Save private key as PuttyKey.ppk
    18. Configure Putty.
      Connection->SSH->AUTH, set C:\EC2\PuttyKey.ppk
    19. Get URL
    20. ec2-describe-instances

      Instance: i-9054cdf2
      URL: ec2-174-129-182-96.compute-1.amazonaws.com
      Url works in browser (Fedora core test page).

    21. Login with Putty. Works.
    22. Shut down.
    23. ec2-terminate-instances i-9054cdf2

      Successfully shut down.

    Interestingly, did not see any activity on the EC2 AWS Management Console. The instance was not loaded on the web console.

    Total time to finish tutorial: 3.5 hours.

    Posted in Parallel, Programming, Uncategorized | Tagged: , , , | Leave a Comment »

    Programming challenge: Python+EC2

    Posted by BenjaminMarcusJones on January 2, 2012

    Objective:

    1. Spawn a windows (image) on Amazon EC2 cloud
    2. Create a basic “hello world” client/server communication script
    3. Load the script onto the image
    4. Send a message to cloud, receive message back

    Posted in Buidling simulation, Programming | Tagged: , , , | Leave a Comment »

    Meta-Heuristic packages and platforms

    Posted by BenjaminMarcusJones on December 28, 2011

    LinkTitleLanguageActivity
    pyevolve PythonActive
    Source at Github

    Documentation at SourceForge

    Author's Blog

    A blog post, getting started
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vitae nisi non est mattis pretium ut non nisi. Aliquam erat volutpat. Vivamus varius arcu in enim euismod dapibus. Donec blandit molestie ligula ac faucibus. Morbi eget sapien vitae purus molestie fermentum sed eu quam. Vestibulum nec neque a neque elementum adipiscing. Praesent vel erat mauris, id mattis turpis. Mauris tincidunt ante in orci mattis fringilla. Donec gravida sem ac leo hendrerit a imperdiet lectus tincidunt. Maecenas eget nunc et ante rhoncus iaculis a in ipsum. Curabitur semper dolor ullamcorper sem auctor et facilisis lacus cursus.
    A Multilevel Parallel Object-Oriented Framework for:

    Design Optimization
    Parameter Estimation
    Uncertainty Quantification
    Sensitivity Analysis

    Posted in Programming | Tagged: , , | Leave a Comment »

    Simulation Parallelization Platforms

    Posted by BenjaminMarcusJones on December 28, 2011

    Draft post

    LinkTitleLanguageActivity
    Distributed Evolutionary Algorithms in PythonPythonActive
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vitae nisi non est mattis pretium ut non nisi. Aliquam erat volutpat. Vivamus varius arcu in enim euismod dapibus. Donec blandit molestie ligula ac faucibus. Morbi eget sapien vitae purus molestie fermentum sed eu quam. Vestibulum nec neque a neque elementum adipiscing. Praesent vel erat mauris, id mattis turpis. Mauris tincidunt ante in orci mattis fringilla. Donec gravida sem ac leo hendrerit a imperdiet lectus tincidunt. Maecenas eget nunc et ante rhoncus iaculis a in ipsum. Curabitur semper dolor ullamcorper sem auctor et facilisis lacus cursus.
    DEXEN - Distributed Execution Environment
    (formerly evo-devo-design)
    PythonLast activity: Jul 31, 2011
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus vitae nisi non est mattis pretium ut non nisi. Aliquam erat volutpat. Vivamus varius arcu in enim euismod dapibus. Donec blandit molestie ligula ac faucibus. Morbi eget sapien vitae purus molestie fermentum sed eu quam. Vestibulum nec neque a neque elementum adipiscing. Praesent vel erat mauris, id mattis turpis. Mauris tincidunt ante in orci mattis fringilla. Donec gravida sem ac leo hendrerit a imperdiet lectus tincidunt. Maecenas eget nunc et ante rhoncus iaculis a in ipsum. Curabitur semper dolor ullamcorper sem auctor et facilisis lacus cursus.
    The DAKOTA ProjectC++Active
    A Multilevel Parallel Object-Oriented Framework for:

    Design Optimization
    Parameter Estimation
    Uncertainty Quantification
    Sensitivity Analysis

     

    Posted in Programming | Tagged: , , | Leave a Comment »

    Automated reporting with Latex and Jinja2 templating

    Posted by BenjaminMarcusJones on September 27, 2011

    Work in progress for the following workflow pipeline

    Excel / Text / Etc… -> Python processing -> Jinja2 templating -> Latex code -> PDF!

    Original idea of using Django instead of Jinja2 has problems, due to the use of curly braces by Django syntax. This obviously causes problems with Latex syntax, resulting in templates that are quite awkward to create.

    Still haven’t implemented a final soluation, but here are some online solutions to consider.

    1. e6h blog post: I like this template so far, it’s clear what a \VAR{} is, but I’m surprised to see the curly braces again.

    http://e6h.de/post/11/

    # The environment
    latex_jinja_env = jinja2.Environment(
    block_start_string = '\BLOCK{',
    block_end_string = '}',
    variable_start_string = '\VAR{',
    variable_end_string = '}',
    comment_start_string = '\#{',
    comment_end_string = '}',
    line_statement_prefix = '%-',
    line_comment_prefix = '%#',
    trim_blocks = True,
    autoescape = False,
    )
    

    1. Flask blog post: I don’t really like the multiple bracket syntax, but the Tex filtering is a great addition!

    http://flask.pocoo.org/snippets/55/

    # The environment
    texenv = app.create_jinja_environment()
    texenv.block_start_string = '((*'
    texenv.block_end_string = '*))'
    texenv.variable_start_string = '((('
    texenv.variable_end_string = ')))'
    texenv.comment_start_string = '((='
    texenv.comment_end_string = '=))'
    texenv.filters['escape_tex'] = escape_tex
    

     

    # Filter 
    LATEX_SUBS = (
    (re.compile(r'\\'), r'\\textbackslash'),
    (re.compile(r'([{}_#%&$])'), r'\\\1'),
    (re.compile(r'~'), r'\~{}'),
    (re.compile(r'\^'), r'\^{}'),
    (re.compile(r'"'), r"''"),
    (re.compile(r'\.\.\.+'), r'\\ldots'),
    )
    
    def escape_tex(value):
    newval = value
    for pattern, replacement in LATEX_SUBS:
    newval = pattern.sub(replacement, newval)
    return newval
    

    Posted in Automated Reporting, Challenge, Programming, Research philosophy | Tagged: , , , | Leave a Comment »

    Parallel execution of a Batch of simulation runs

    Posted by BenjaminMarcusJones on April 7, 2011

    Here’s a video demonstrating creation and parallel (local machine) execution of a design space containing 36 simulation runs using self developed code in Python and TRNSYS 17. The module will simulate the batch of 36 using a specified maximum CPU (100% here) and number of parallel threads (4 here). This concept can be generalized to any set ‘A’ containing ‘m’ design variables, each ‘Ai’ having a length ‘ni’, and executed using any “input file” -> “simulate.EXE” style simulation software (“Embarrassingly parallel”).

    Code is object oriented, using Python 2.6, with standard libraries.

    I also wanted to investigate how my hyperthreaded intel quad core machine would handle parallelization of TRNSYS runs.

    We can see that above 4-5 cores (Threads-axis), overall performance remains constant (batch completion time remains 50s, z-axis). Hyperthreading cores therefore have no benefit, even though your TRNEXE thread will show 12.5% CPU time. Finally, the “Batch” module can specify a ceiling on CPU usage (if ie/ you want to use your machine while TRNSYS executes).

    Open question: Can TRNSYS run COMPLETELY in the background? I don’t need to see these “calculation” windows all the time…

    More info to follow, just a quick post for now!

    Posted in Buidling simulation, Programming | Tagged: , , , | 2 Comments »

     
    This site is protected by Comment SPAM Wiper.