tag:blogger.com,1999:blog-17189212944351408102024-03-08T03:33:32.761-08:00The Born-again SysadminMeh.Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.comBlogger187125tag:blogger.com,1999:blog-1718921294435140810.post-64236582092771587952023-07-11T08:12:00.008-07:002023-07-11T10:35:51.280-07:00Using SSM Automation as a makeshift web form for invoking Lambda Functions (and Step Functions, too!)<h2>What's this article about?</h2><p>It's about <b><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html">AWS Systems Manager Automation</a>. </b>This product can be seen as a poor man's Step Function which includes a basic and easy-to-use web frontend that makes it trivial to execute operational tasks that could otherwise be tedious.</p><h2 style="text-align: left;">The need for simple web forms</h2><p style="text-align: left;">Many of our IT-related processes need to interact with various internal web services. The logic to speak with these services is implemented using Lambda functions. </p>Each of these Lambda functions expect a curated JSON payload with multiple keys. Building the payloads and invoking the functions from the web console or the CLI is not only time-consuming, but complex. <br /><p style="text-align: left;">Since these tasks are repetitive and not completely automatable, we would like to be able to offload them to Level 1 support personnel, while limiting training and reducing data entry errors.</p><p style="text-align: left;">Doing so requires us to provide to IT support staff a web-based interface that is more user-friendly than AWS's built-in Lambda web console. A simple web form that can validate<b> </b>input data upfront, provide user feedback, and format JSON payloads automatically would fit the bill, right?</p><p style="text-align: left;">The question arises, how can one make such a web form quickly, and, if possible, define it with IaC?</p><div><h2 style="text-align: left;">SSM Automation to the rescue!</h2><p></p><p>The most obvious answer has been to use <b><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-automation.html">SSM Automation</a> </b>for this purpose.<br /><br />Simply put, an SSM Automation can be seen as a much simpler Step Function geared towards low volume and interactive use. <br /><br />From what I can tell, this Systems Manager feature was originally designed to automate tasks related to EC2 instance management. But I found that it can invoke <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-lamb.html">Lambda</a> and <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-action-executeStateMachine.html">Step Functions</a> too and it can also call arbitrary AWS API functions if you require.</p><h3 style="text-align: left;">Key points</h3><p style="text-align: left;">An SSM Automation is defined in a standard SSM document. The schema is currently at <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/documents-schemas-features.html">revision 0.3</a>.</p><p style="text-align: left;">When the automation is executed, a <b>basic web form </b>is automatically generated based on parameters you've set in the document. Note that this form is not extremely customisable. </p><p style="text-align: left;">This web form contains <b>input parameters</b> that support input validation which are similar to CloudFormation parameters (i.e. AllowValues, AllowedPattern, etc). These parameters can then be formatted in a structured payload when invoking your function.</p><p style="text-align: left;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgdXsPl9ZohpLzGG2AA5siGK6thmu_aaj5UdkXrlujaQqwUUKlNzHgFuPnnF7scDYU8p9rAuGCPdJnMKRzGrhTnx6bx96UIb56Y7NMZUve-bPeumw4AwP0J6YXjwhQvGH71qtn6kNUOs6oi1xtKMVaWuYtbzUbHDs5JkKhsrj4biP0dCj5LdwLd2a2oOVM" style="margin-left: auto; margin-right: auto;"><img alt="" data-original-height="358" data-original-width="903" height="127" src="https://blogger.googleusercontent.com/img/a/AVvXsEgdXsPl9ZohpLzGG2AA5siGK6thmu_aaj5UdkXrlujaQqwUUKlNzHgFuPnnF7scDYU8p9rAuGCPdJnMKRzGrhTnx6bx96UIb56Y7NMZUve-bPeumw4AwP0J6YXjwhQvGH71qtn6kNUOs6oi1xtKMVaWuYtbzUbHDs5JkKhsrj4biP0dCj5LdwLd2a2oOVM" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The web form is generated based on your parameter settings.</td></tr></tbody></table></p><p style="text-align: left;">Logs from your Lambda functions are <b>much</b> easier to consume in the web interface compared to going directly in Cloudwatch. Each step is logged as if it was invoked with the "tail" output option as the command-line, and saved in the execution log.</p><br /><div><p style="text-align: left;"><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhBKsTHn1fMMHz20HT529EVIGgxuL-35sLp-9C_oC8GSt8vyI0jPSaNTYIQbIQH2si9Jk5_-U2pLDdgBDDu-KPYPRuQzv2pO3KejaO-As60PsIOtX3ZIn5fNE6Q_b-zTPFmVqpIQ4Ty5Mve16NPwArEMUD9okJsDx9Dr8auab8183hlyZ6X8a5zWSpQo5M" style="margin-left: auto; margin-right: auto;"><img alt="" data-original-height="757" data-original-width="1138" height="213" src="https://blogger.googleusercontent.com/img/a/AVvXsEhBKsTHn1fMMHz20HT529EVIGgxuL-35sLp-9C_oC8GSt8vyI0jPSaNTYIQbIQH2si9Jk5_-U2pLDdgBDDu-KPYPRuQzv2pO3KejaO-As60PsIOtX3ZIn5fNE6Q_b-zTPFmVqpIQ4Ty5Mve16NPwArEMUD9okJsDx9Dr8auab8183hlyZ6X8a5zWSpQo5M" width="320" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">The automation log lets you check the output of your lambda functions quickly, no need to search into countless log streams in Cloudwatch!</td></tr></tbody></table></p></div><div>Something cannot be done directly in an automation step? You can use <b>inline scripts</b> in Python or Powershell. There is no need for you to host these scripts in Lambda, it is taken care of automatically.<p style="text-align: left;">Oh, and everything can be defined using <b>CloudFormation</b> or the <b>CDK</b>.</p><p style="text-align: left;">There are many examples that AWS have already done themselves, and that you can see for yourself, such as <a href="https://ca-central-1.console.aws.amazon.com/systems-manager/automation/execute/AWS-CreateRdsSnapshot">this one</a>.</p><div><h3 style="text-align: left;">Some caveats</h3><p style="text-align: left;">The automation language is not as complete or as feature-rich as the one used with Step Functions and, I can assume, is not intended to be.</p><p style="text-align: left;">There is throttling in place. Don't use this for high-volume transactions.</p></div><div><h2 style="text-align: left;">Wrap-up</h2><p style="text-align: left;">Using the SSM Automation web interface, there is no longer a need to prepare and curate JSON payloads manually and invoking Lambda or Step Functions becomes very easy to support staff.</p><p style="text-align: left;">I hope this quick article helped you get an idea of how SSM Automation can be of help with this use case.</p></div></div></div>Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-29035002959195858682021-07-29T11:11:00.005-07:002021-07-29T11:17:02.778-07:00Using cfn_nag with the CDK<p><i></i></p><blockquote><i>Nota bene: As I was writing this post, I noticed that CDK Labs has started work on <a href="https://github.com/cdklabs/cdk-nag">cdk-nag</a> which will let you analyze your work directly from within your code.</i></blockquote><p></p><p>We've been using Stelligent's excellent <a href="https://github.com/stelligent/cfn_nag">cfn_nag </a>for over a year to ensure that our homemade CloudFormation templates are following minimal security best practices. As we've begun transitioning to the CDK, we now need to do similar analysis on synthesized templates generated by the CDK.</p><p>First of all, up until now, it seems that cfn_nag is perfectly capable of analyzing templates synthesized from the CDK. The templates themselves are harder to read due to their nature, but the tool still works.</p><p><br /></p><h3 style="text-align: left;">Running cfn_nag on synthesized templates</h3><div><br /></div><div>Running cfn_nag in a CDK workflow is fairly simple, just synthesize the template with "<b>cdk synth</b>" then run cfn_nag_scan on your template as you would do if creating it by hand, i.e. "<b>cfn_nag_scan -i cdk.out/mytemplate.json</b>". There is no more to it.</div><div><br /></div><div><br /></div><h3 style="text-align: left;">Adding exceptions in your CDK code</h3><p>A small challenge for me was to find a way to embed the cfn_nag exceptions in the synthesized output. Thanks to <a href="https://stackoverflow.com/questions/57615473/how-to-add-metadata-to-iam-policy-using-aws-cdk">Yan Xiao's stackoverflow post</a>, I've been able to do this quite easily.</p><p>Please note that I'm not a typescript expert (or javascript, for the matter) so I'll show you how I've done it, but keep in mind that it might not be the best way.</p><p><br /></p><h4 style="text-align: left;">Using an array to hold the exceptions</h4><p>First, define empty arrays that will contain the exceptions for each of the resources in your code, for example:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div> <span style="color: #6a9955;">// Array containing cfn_nag exceptions for the Lambda Function</span></div><div> <span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">cfnnag_LambdaFunctionRulesToSupress</span> = [];</div><div> <span style="color: #6a9955;">// Array containing cfn_nag exceptions for the Bucket</span></div><div> <span style="color: #569cd6;">var</span> <span style="color: #9cdcfe;">cfnnag_BucketRulesToSupress</span> = [];</div></div><p>As your code flows, add the exceptions that you need in the appropriate array, for example, if you need to add an exception to your bucket, do this:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div> <span style="color: #9cdcfe;">cfnnag_BucketRulesToSupress</span>.<span style="color: #dcdcaa;">push</span>({</div><div> <span style="color: #ce9178;">"id"</span><span style="color: #9cdcfe;">:</span> <span style="color: #ce9178;">"W35"</span>, </div><div> <span style="color: #ce9178;">"reason"</span><span style="color: #9cdcfe;">:</span> <span style="color: #ce9178;">"No need for logging on this Bucket"</span></div><div> });</div></div><p>Note here that I'm appending new exceptions to the array using <b>push</b>. So just push as many of these exceptions as you need.</p><p><br /></p><h4 style="text-align: left;">Applying the exceptions as CloudFormation Metadata</h4><p>Once you've finished defining the resources, you'll need to add the exceptions as CloudFormation metadata using the Level 1 properties. Here are a two examples.<br /><br />1. This adds the exceptions as metadata to your bucket, assuming that you've pushed some exceptions in the array beforehand:</p><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; line-height: 19px; white-space: pre;"><div style="line-height: 19px;"><div style="line-height: 19px;"><div> <span style="color: #569cd6;">const</span> <span style="color: #4fc1ff;">S3CfnBucket</span> = <span style="color: #569cd6;">this</span>.<span style="color: #4fc1ff;">S3Bucket</span>.<span style="color: #4fc1ff;">node</span>.<span style="color: #9cdcfe;">defaultChild</span> <span style="color: #c586c0;">as</span> <span style="color: #4ec9b0;">s3</span>.<span style="color: #4ec9b0;">CfnBucket</span>;</div><div> <span style="color: #c586c0;">if</span> (<span style="color: #9cdcfe;">cfnnag_BucketRulesToSupress</span>.<span style="color: #9cdcfe;">length</span> > <span style="color: #b5cea8;">0</span> ) {</div><div> <span style="color: #569cd6;">this</span>.<span style="color: #4fc1ff;">S3CfnBucket</span>.<span style="color: #4fc1ff;">cfnOptions</span>.<span style="color: #9cdcfe;">metadata</span> = { </div><div> <span style="color: #ce9178;">"cfn_nag"</span><span style="color: #9cdcfe;">:</span> {</div><div> <span style="color: #ce9178;">"rules_to_suppress"</span><span style="color: #9cdcfe;"> :</span> <span style="color: #9cdcfe;">cfnnag_BucketRulesToSupress</span></div><div> }</div><div> }</div><div> }</div></div></div></div><p>2. When adding policies directly to a bucket using the <a href="https://docs.aws.amazon.com/cdk/api/latest/docs/@aws-cdk_aws-s3.Bucket.html#addwbrtowbrresourcewbrpolicypermission">addToResourcePolicy</a> method, things get more tricky as the CDK will not embed these policies in your bucket, but will create a separate <b>AWS::S3::BucketPolicy</b> resource. So if you have specific cfn_nag exceptions to apply to your policies, do this to add them as metadata to the resource :<br /></p><div style="line-height: 19px;"><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> <span style="color: #569cd6;">const</span> <span style="color: #4fc1ff;">S3CfnbucketPolicy</span> = <span style="color: #569cd6;">this</span>.<span style="color: #4fc1ff;">S3Bucket</span>.<span style="color: #9cdcfe;">policy</span>?.<span style="color: #4fc1ff;">node</span>.<span style="color: #9cdcfe;">defaultChild</span> <span style="color: #c586c0;">as</span> <span style="color: #4ec9b0;">s3</span>.<span style="color: #4ec9b0;">CfnBucketPolicy</span>;</div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> <span style="color: #c586c0;">if</span> (<span style="color: #9cdcfe;">cfnnag_BucketPolicyRulesToSupress</span>.<span style="color: #9cdcfe;">length</span> > <span style="color: #b5cea8;">0</span> ) {</div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> <span style="color: #4fc1ff;">S3CfnbucketPolicy</span>.<span style="color: #4fc1ff;">cfnOptions</span>.<span style="color: #9cdcfe;">metadata</span> = { </div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> <span style="color: #ce9178;">"cfn_nag"</span><span style="color: #9cdcfe;">:</span> {</div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> <span style="color: #ce9178;">"rules_to_suppress"</span><span style="color: #9cdcfe;"> :</span> <span style="color: #9cdcfe;">cfnnag_BucketPolicyRulesToSupress</span></div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> }</div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> }</div><div style="background-color: #1e1e1e; color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"> }</div><div style="color: #d4d4d4; font-family: Consolas, "Courier New", monospace; font-size: 14px; white-space: pre;"><span style="color: black; font-family: "Times New Roman"; font-size: medium; white-space: normal;"><br /></span></div><div>So here it goes. Hope this helps.</div></div>Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-26108541806112025912019-11-13T12:06:00.002-08:002019-11-13T12:07:24.796-08:00Cross-account sharing of a PrivateLink endpoint using Private Hosted Zones and CloudFormation<h2>
Introduction</h2>
It is possible to concentrate all your <a href="https://docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html">PrivateLink endpoints</a> in one account, then share them with other accounts and access them through a Transit Gateway.<br />
<br />
This reduces the consumption of private IPs and makes everything cleaner. You also do not have to pay a <a href="https://aws.amazon.com/privatelink/pricing/">hourly fee</a> for these endpoints in each of your accounts, although you still have to consider the transit fees involved to bring the data in another VPC.<br />
<br />
This is done using Route53 and Private Hosted Zones (PHZs). James Levine's post <a href="https://aws.amazon.com/blogs/networking-and-content-delivery/integrating-aws-transit-gateway-with-aws-privatelink-and-amazon-route-53-resolver/">Integrating AWS Transit Gateway with AWS PrivateLink and Amazon Route 53 Resolver</a> explains very clearly how you can achieve this. I'll spare the details but basically, this lets you override the DNS addresses of the endpoints within your accounts to point to your private address instead of the public one.<br />
<br />
Go read James' article first, then come back here for implementation details.<br />
<h2>
A sample use case</h2>
The use case that made me do this initially is <a href="https://aws.amazon.com/systems-manager/">AWS Systems Manager</a>. I wanted to be able to use its Session Manager feature to open interactive sessions on EC2 instances, in multiple accounts. Since I wanted to prevent routing SSM data through the internet, combined with the fact that it required many VPC endpoints, I decided to concentrate them in one account.<br />
<br />
As <a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html">documented</a>, four VPC interface endpoints (i.e. PrivateLink endpoints) are needed for this: <i>ssm</i>, <i>ssmmessages</i>, <i>ec2</i> and <i>ec2messages</i>. There is also a fifth endpoint for S3, but that one is a gateway endpoint and it needs to be defined in each of your accounts.<br />
<br />
When an EC2 instance tries to communicate with the <i>ssm</i> endpoint, its agent looks up that endpoint's DNS address and by default, it gets the public IP address for your region. For example:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ nslookup ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Name: ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Address: 52.94.100.144</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">But what do you do if you have defined a PrivateLink endpoint for <i>ssm.ca-central-1.amazonaws.com</i> in another account, and you wish to use it through a peering connection or a Transit Gateway? James explains how to configure a DNS hosted zone to fool everything in that VPC into using a private address. A lookup in this EC2 instance will then give this result:</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ nslookup ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Name: ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Address: 192.168.0.10</span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
where 192.168.0.10 is the private IP address assigned to the VPC endpoint.<br />
<br />
This works because that DNS record is, in fact, an alias record on the regional address of your endpoint. For example, <i>ssm.ca-central-1.amazonaws </i>is aliased to <i>vpce-0123456789abcdef-01234abcd.ssm.ca-central-1.vpce.amazonaws.com.</i><br />
<br />
If you went further and defined your VPC in multiple AZs, you should have multiple addresses :</div>
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">$ nslookup ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-size: x-small;"><br /></span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Name: ssm.ca-central-1.amazonaws.com</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Address: 192.168.0.10, 192.168.1.10</span><br />
<br />
N.B. I'm still not sure what the impact of a failure in one AZ is in this scenario as I'm well-aware that using DNS as a failover mechanism isn't great and can involve timeouts. I hope that any unavailable IP address will be removed dynamically from the regional address... I don't have the answer to this one.<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<h2>
Configuring a PrivateLink endpoint with a PHZ in CloudFormation</h2>
<span style="font-size: small; font-weight: 400;">Here are the three resources you need to put in your CF template to deploy an endpoint and a private hosted zone. Once you have these in place, it is a matter of repeating the same code for </span><i style="font-size: medium; font-weight: 400;">ssmmessages</i><span style="font-size: small; font-weight: 400;">, </span><i style="font-size: medium; font-weight: 400;">ec2</i><span style="font-size: small; font-weight: 400;"> and </span><i style="font-size: medium; font-weight: 400;">ec2messages</i><span style="font-size: small; font-weight: 400;">.</span><br />
<br />
<b>
Security Group</b><br />
<div>
The first thing you need is to define a security group to make your endpoint accessible on the port it needs (usually TCP/443):</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ssmendpointSG:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Type: AWS::EC2::SecurityGroup</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Properties:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GroupDescription: "SG for SSM endpoint"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GroupName: "SSMSG"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SecurityGroupIngress:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - CidrIp: 10.10.0.0/16</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> IpProtocol: tcp</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> FromPort: 443</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ToPort: 443</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Tags:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - Key: Name</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Value: "My SSM Endpoint SG"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> VpcId:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Fn::ImportValue: "VPC-id-outputvariable-from-another-template"</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Note here that I've imported the VPC ID using an output variable that comes from another CF template. You can hardcode it or input it as a parameter if you prefer.</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<b>
PrivateLink Endpoint</b></div>
<div>
Then, you define the PrivateLink endpoint itself:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ssmendpointVPC:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Type: AWS::EC2::VPCEndpoint</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Properties:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> VpcEndpointType: Interface</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ServiceName: !Sub com.amazonaws.${AWS::Region}.ssm</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SecurityGroupIds:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - !Ref ssmendpointSG</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> SubnetIds:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - Fn::ImportValue:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> !Sub "AZ1-subnet-id-outputvariable-from-another-template"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - Fn::ImportValue:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> !Sub "AZ2-subnet-id-outputvariable-from-another-template"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> VpcId:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Fn::ImportValue:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> !Sub "VPC-id-outputvariable-from-another-template"</span></div>
</div>
<div>
<br /></div>
<span style="font-family: inherit;"><b>Private hosted zone</b></span><br />
<div>
<span style="font-family: inherit;">As for the private zone, these two entries need to be configured (you'll need to replace the region with yours):</span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> phzssmcacentral1amazonawscom:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Type: AWS::Route53::HostedZone</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Properties:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Name: ssm.ca-central-1.amazonaws.com</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> VPCs:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> - VPCId:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Fn::ImportValue: "</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">VPC-id-outputvariable-from-another-template</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> VPCRegion: "ca-central-1"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> phzaliasrecordssmcacentral1amazonawscom:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Type: AWS::Route53::RecordSet</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Properties:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> AliasTarget:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> HostedZoneId: !Select [ '0', !Split [ ':', !Select [ '0', !GetAtt </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ssmendpointVPC</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">.DnsEntries ]]]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> DNSName: !Select [ '1', !Split [ ':', !Select [ '0', !GetAtt </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ssmendpointVPC</span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">.DnsEntries ]]]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> HostedZoneId: !Ref </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">phzssmcacentral1amazonawscom</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Name: ssm.ca-central-1.amazonaws.com.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Type: A</span></div>
</div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: inherit;">Note the multiple selectors under <i>AliasTarget</i>. The combination of these selectors extracts <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-vpcendpoint.html">specific fields from AWS::EC2::VPCEndpoint</a> that are made available as attributes once CloudFormation deploys the endpoint, namely:</span><br />
<br />
<ul>
<li><span style="font-family: inherit;">The hosted zone ID for the endpoint</span></li>
<li><span style="font-family: inherit;">The regional DNS address of the endpoint (as opposed to the one pointing to the AZ itself)</span></li>
</ul>
</div>
<div>
<br /></div>
<h2>
Sharing a PHZ across accounts</h2>
<div>
Once the PHZ is deployed, you need it to share it with your accounts. Unfortunately, you cannot do this with CloudFormation. The procedure is explained in the KB article <a href="https://aws.amazon.com/premiumsupport/knowledge-center/private-hosted-zone-different-account/">How do I associate a Route 53 private hosted zone with a VPC on a different AWS account?</a>. It explains how to do this using the CLI.</div>
<div>
<br /></div>
<div>
Good luck.</div>
<div>
<br /></div>
<div>
<br /></div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-61144602175600314312019-10-24T16:23:00.000-07:002019-10-25T06:05:31.491-07:00Deploying a cross-account Transit Gateway using CloudFormation<br />
<h3>
<br />Introduction</h3>
I've decided to automate the deployment of a Transit Gateway using CloudFormation.<br />
<br />
I'll show you here how I did it, but be advised that it is currently not possible to do complex configurations on a TGW using CloudFormation. You will need to do some tasks manually, at least one which can only be done with the AWS CLI.<br />
<br />
<h3>
First, some caveats</h3>
Now there are a few caveats you need to be aware of before using CloudFormation to deploy a Transit Gateway:<br />
<br />
<ul>
<li>At this time, <a href="https://github.com/aws-cloudformation/aws-cloudformation-coverage-roadmap/issues/61">there are no attributes whatsoever that can be extracted from GetAtt</a>, which means you can't extract its ARN, default route table ID, and others and use them later in your template. </li>
<li>Every (and I mean <i>every</i>) property change requires a <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-replacement">replacement</a>, which is a big deal.</li>
<ul>
<li>This means that doing something as simple as trying to change a tag on your Transit Gateway using CloudFormation will cause downtime, as it requires that you first remove any attachments and dependencies before being able to update the stack.</li>
<li>It also means that the ID and ARN of the TGW itself will change once it is replaced, which requires lots of planning: any dependents that refer to these identifiers will need to be reconfigured.</li>
</ul>
</ul>
<br />
<h3>
tgw-main.yml : Deploying the Transit Gateway</h3>
<div>
Deploying a TGW is fairly straightforward:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Resources:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> mytgw:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Type: 'AWS::EC2::TransitGateway'</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Properties:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> AutoAcceptSharedAttachments: enable</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Tags:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - Key: Name</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Value: "My Transit Gateway"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Outputs:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> outmytgw:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Description: TGW ID</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Value: !Ref mytgw</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Export:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Name: "mytgw-id"</span><span style="font-family: "courier new" , "courier" , monospace;"></span></div>
<div>
<br /></div>
<div>
<br />
I've set <i>AutoAcceptSharedAttachments</i> to <b>enable</b> to prevent having to accept VPC attachments manually, as they will be done later.<br />
<br /></div>
<div>
I've also added an output variable. It is set so that I can then reference the TGW ID from other stacks, namely the VPC-related stacks that will attach themselves to the TGW. I suggest you export it with the name <span style="font-family: "courier new" , "courier" , monospace;">!Sub "${AWS::StackName}-mytgw-id" </span><span style="font-family: inherit;">if you prefer prefixing it with the stack name.</span></div>
<div>
<br /></div>
<div>
Caution: Whatever you do, be sure to understand all the properties of <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-ec2-transitgateway.html">AWS::EC2::TransitGateway</a> and their implications. As I said earlier, you <b>cannot</b> change any of them once it's deployed without replacing the TGW, and removing all the dependencies below (and possibly more).<br />
<br /></div>
<div>
<h3>
tgw-ram.yml: Sharing the TGW across different accounts (optional)</h3>
</div>
<div>
If you need to attach to the TGW from a VPC in another account, you first need to use Resource Access Manager (RAM) to share it between your accounts.</div>
<div>
<br /></div>
<div>
This cannot be done in the previous stack (<b>tgw-main.yml</b>); sharing the TGW requires getting its ARN and as explained in the <i>Caveats</i> section, there is no way do to that from CloudFormation. To my knowledge, it's not available in the portal either. Therefore, you first need to extract the ARN using the AWS CLI:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ aws ec2 describe-transit-gateways</span></div>
<div>
<br />
This will show you the ARN, such as:<br />
<span style="font-family: "courier new" , "courier" , monospace;">arn:aws:ec2:xx-xxxxx-x:yyyyyyyyyyy:transit-gateway/tgw-zzzzzzzzzzz</span></div>
<div>
<br />
Where:<br />
<br />
<ul>
<li>xx-xxxx-x: The AWS region where the TGW is located</li>
<li>yyyyyyyy: The account number that hosts the TGW</li>
<li>tgw-zzzzz: The TGW ID.</li>
</ul>
</div>
<div>
Then, you can build your RAM template like this:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Resources:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> sharemytgw:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Type: "AWS::RAM::ResourceShare"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Properties:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Name: "My TGW RAM Share"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ResourceArns: arn:aws:ec2:my-aws-region:my-aws-account:transit-gateway/tgw-my-tgw-id</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Principals:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "first_account_number"</span><br />
<span style="font-family: "courier new", courier, monospace; white-space: pre;"> </span><span style="font-family: "courier new", courier, monospace;">- "second_account_number"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Tags:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - Key: "Name"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Value: "My TGW RAM Share"</span><br />
<br /></div>
<div>
N.B. I actually use a <a href="https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/parameters-section-structure.html">parameter</a> for <span style="font-family: "courier new" , "courier" , monospace;">ResourceArns</span>, so I don't have to hardcode the ARN in there. I've left it out to keep things simple.</div>
<div>
<br />
Once this template is run, you need to go manually inside each account and accept the Resource Access Manager invite. There is no way, to my knowledge, of doing this within CloudFormation.<br />
<br /></div>
<h3>
tgw-vpc-attach.yml: Attaching a VPC to the TGW</h3>
<div>
Assuming you already have a CloudFormation Template to deploy your VPCs, it is then a matter of adding this code to have them attach to the TGW:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Resources:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> vpctgwattach:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Type: 'AWS::EC2::TransitGatewayAttachment'</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Properties:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> TransitGatewayId:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Fn::ImportValue:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> !Sub "mytgw-id"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> VpcId: !Ref myvpc</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> SubnetIds: <span style="white-space: pre;"> </span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - !Ref mysubnetAZ1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - !Ref mysubnetAZ2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Tags:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - Key: Name</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Value: "VPC TGW attachment"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Outputs:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> outvpctgwattach:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Description: VPC TGW Attachment ID</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Value: !Ref vpctgwattach</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Export:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Name: "vpctgwattach-id"</span><span style="font-family: "courier new" , "courier" , monospace;"></span><br />
<br /></div>
<div>
See here that I refer to the output variable defined previously in tgw-main.yml in order to get the ID of the TGW (without the stack name, but this is up to you).<br />
<br />
This is for a VPC located in the same account as the TGW; note that referencing CloudFormation output variables doesn't work across accounts, the TGW ID can then be hardcoded. There are workarounds, but from what I've seen, they involve Lambda functions and I prefer avoiding this for the moment.<br />
<br />
The TGW needs to be attached to a subnet in each of the AZs that your VPC spans to. It doesn't matter which subnet you pick these AZs, but you need one. The attachment creates a "secret" endpoint that consumes an IP address in each subnet and all packets that go to the TGW will be routed through it.</div>
<div>
<br /></div>
<div>
While it could be possible to attach to that VPC directly from<b> tgw-main.yml</b>, I've decided not to do this, as I prefer <i>not</i> having to modify the main TGW template when adding new VPCs. It must also be done from within the account that owns the VPC, so I prefer keeping the attachment business out of the main template.</div>
<div>
<br /></div>
<h3>
tgw-defaultroutetable.yml: Adding entries to the default route table</h3>
<div>
There is no way to extract the ID of the default route table from CloudFormation, so you first need to extract it using the CLI or the Portal. The value is labeled as <i>tgw-rtb-xxxxxx</i> where xxxxxx is the Transit Gateway ID.<br />
<br /></div>
<div>
Then, adding a new route is a matter of invoking AWS::EC2::TransitGatewayRoute while referring to the route table ID. I suggest you use a parameter for the route table ID, to your leisure.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Resources:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> tgwdefaultroutetableentry1:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Type: AWS::EC2::TransitGatewayRoute</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Properties:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> DestinationCidrBlock: 0.0.0.0/0</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> TransitGatewayAttachmentId:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Fn::ImportValue:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> !Sub "mytgw-id"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> TransitGatewayRouteTableId: </span><span style="font-family: "courier new" , "courier" , monospace;">"tgw-rtb-xxxxxxxxxxxxx"</span><br />
<br />
<span style="font-family: inherit;">Notice here that I've used the export variable </span><span style="font-family: "courier new" , "courier" , monospace;">mytgw-id </span><span style="font-family: inherit;">to identify my transit gateway.</span><br />
<span style="font-family: inherit;"><br /></span></div>
<div>
<h3>
Wrapping it all up</h3>
</div>
<div>
Deploying a TGW using CloudFormation and sharing it across accounts is a multi-step process:<br />
<br />
<ul>
<li>Deploy the TGW using <b>tgw-main.yml</b></li>
<li>Get the ARN manually using the AWS CLI (or some other way), then share it with with other accounts using <b>tgw-share.yml</b></li>
<li>Go into each account and accept the share invitation.</li>
<li>Create a template to attach the VPCs named <b>tgw-vpc-attach.yml </b>or better, add the code of that template in your current VPC template(s).</li>
<li>Get the default route table ID using the portal (or some other way) and add route entries to the TGW using yet another template named <b>tgw-routetable.yml</b></li>
</ul>
</div>
<div>
<br />
That's about it.<br />
<br /></div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-63175057812306488652019-07-31T18:08:00.000-07:002019-07-31T18:17:32.762-07:00Update and thoughts on Ansible for cloud automationExcept for a few posts here and there, there hasn't been much really useful content in this blog in almost eight years! I think an update is in order.<br />
<br />
I started this blog initially to target mostly HP-UX as I was feeling comfortable enough to post on various subjects on this operating system, and few, if anybody, blogged on HP-UX outside of the official channels, making this niche blog relevant.<br />
<br />
Then I moved on in 2010. Since then, HP-UX itself as a platform has moved on itself, with fewer and fewer systems running. And in the years that followed, I'll be the first to admit that it has not been easy to find a subject on which I felt good enough to blog about.<br />
<br />
This is partly because I could not get a foothold on any particular technology. I've briefly worked as a systems architect, then came back to the technical side in 2014 by keeping Tru64 systems up and running until they got decommissioned (this was in an environment with extremely strict compliance rules -- to be honest, it wasn't very exciting). I then assisted in deploying some Windows servers (!!) in 2015-2016, along with some Red Hat Linux systems, and finally, in 2017, I've got drafted to help upgrading some Solaris 11.3 servers on a few SuperClusters. Okay, <i>drafted</i> is a strong word, it's a terrific and exciting platform, but sorry Solaris, seems to me that you're slowly moving on like HP-UX, too.<br />
<br />
For a year now, I've been working on automating deployments in Azure in a new team. This is a 180 degree turn for a systems administrator, and I like it.<br />
<br />
We're using Ansible to do this, using it to call (somewhat in preferred order):<br />
<br />
<ul>
<li>native Ansible modules (when exist, and also when they don't crash)</li>
<li>AZCLI</li>
<li>REST API calls using azure_rm_resource whenever possible</li>
<li>ARM templates </li>
<li>Powershell (last resort on a Linux host)</li>
</ul>
<br />
Is Ansible great at this job? It's been one year now, and I'm still not sure.<br />
<br />
For starters, it takes a long time to make the code fully bullet-proof and idempotent. Furthermore, while Ansible (especially the modules) makes it easy to expect a desired state for specific Azure resources, it is harder to make a playbook that will take care of not only deploying resources, but reporting differences over time (i.e. drift management) and deleting these resources in the future when they will no longer be needed.<br />
<br />
Terraform has been sugested many times to resolve this, but I haven't looked into it yet. Well, actually I did, but after an hour I still couldn't find out how to print "hello world" so I kind of called it quits, there is so much work to be done that side projects are kind of limited right now.<br />
<br />
AWS seems to have got it right with Cloud Formation and stacks, a feature which, I think, is missing from ARM templates for now as ARM templates seem to be designed to be a one-time thing. I've just learned about stacks today and I'm getting excited.<br />
<br />
To be continued!<br />
<br />
<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-46850613025844057402018-03-05T06:54:00.000-08:002018-03-05T06:54:06.325-08:00Installing Solaris 11.4 beta on a Proliant G4<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzE8JEpce6WbhpQGHNlx9_Q3pKXo_ZryZb15QWPojx0r48R8NbVRv3dtz_jWtSK02Z3iD6m9KYqM5lU-AzXi9d8LndrmEvvJObs5JkbfPpEDxhg1EA0AtgY2Y4UZgE6jAhLtlwLYzRi04/s1600/Capture.PNG" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="299" data-original-width="973" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzE8JEpce6WbhpQGHNlx9_Q3pKXo_ZryZb15QWPojx0r48R8NbVRv3dtz_jWtSK02Z3iD6m9KYqM5lU-AzXi9d8LndrmEvvJObs5JkbfPpEDxhg1EA0AtgY2Y4UZgE6jAhLtlwLYzRi04/s640/Capture.PNG" width="640" /></a></div>
<br />
I've been trying to install Solaris 11.4 beta on an extremely old x86 server, in part because I do not have access to a scratch VMware environment and also to see if I could pull it off.<br />
<br />
I had access to a bunch of unused HP Proliant DL360 G4s. They are listed as <i>reported to work</i> on the <a href="http://www.oracle.com/webfolder/technetwork/hcl/data/systems/details/hewlett-packard/sol_9_09_04/691.html">Hardware Compatibility List</a>, so I said to myself "Why not". So I scavenged memory and CPUs and tried to install the OS.<br />
<br />
I was able to boot the install media using a USB key, but the graphic card didn't seem to be compatible, as I got the message "Compatible fb not found". Specifying <tt style="background-color: white; color: #444444; font-family: Monaco, Courier, "Courier New", monospace; font-size: 12px; font-weight: 700;">-B</tt><span style="background-color: white; font-family: Arial, Helvetica, Luxi-sans, "Nimbus Sans L", sans-serif; font-size: 12px; font-weight: 700;"> </span><tt style="background-color: white; color: #444444; font-family: Monaco, Courier, "Courier New", monospace; font-size: 12px; font-weight: 700;">console=force-text</tt> didn't work, it switched to graphical mode anyway.<br />
<br />
It took multiple tries and reboots to find a combination that worked. I found out that it is possible to install on a serial console. There are GRUB menu entries that let you boot the OS using <i>ttya</i> or <i>ttyb</i>, but they are hidden. I'm not sure how I got into this menu, but I think it was by pressing ESC at the GRUB prompt that gives you 5 seconds before booting the OS.<br />
<br />
I attached a laptop with a serial cable to the server and ran screen in an xterm. I've been able to access the text installer sucessfully and install the OS.<br />
<br />
My system now boots. I'm waiting for my network patch request to come through before continuing.<br />
<br />
I'm especially interested in trying the new <a href="https://blogs.oracle.com/jmcp/solaris-analytics%3a-an-overview">Solaris Analytics</a> interface. I'll keep you posted.<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-55898462843875909532017-05-11T12:07:00.000-07:002017-05-11T12:19:43.842-07:00Revisiting the restricted shellI've been administering Unix boxes since the mid-90s and I've always been told that using restricted shells (rsh, rksh, rbash) was a bad idea because they are easily hackable. Indeed, there are <a href="https://speakerdeck.com/knaps/escape-from-shellcatraz-breaking-out-of-restricted-unix-shells">countless</a> <a href="https://pen-testing.sans.org/blog/2012/06/06/escaping-restricted-linux-shells">known</a> <a href="http://securebean.blogspot.ca/2014/05/escaping-restricted-shell_3.html">methods</a> to get out of a restriced shell: from finding an application that allows a shell escape, to trying to compile your own, to doing clever hacks with the history file.<br />
<br />
I've recently been in a corner case where I was dealing with an embedded product which requires a specific set of commands and also uses some bracket commands that are difficult to wrap with our usual SSH command authenticator. So I decided to revisit using a restricted shell to jail this user and I think I managed to make the jail shatterproof enough.<br />
<div>
<br /></div>
Here is how I did it:<br />
<br />
Create Bob's home directory, but assign it to root:<br />
<span style="font-family: "courier new" , "courier" , monospace;"># mkdir /home/bob</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># chown root:root /home/bob</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># chmod 755 /home/bob</span><br />
<br />
Force a .bashrc and .profile that changes Bob's PATH to a limited set of commands:<br />
<span style="font-family: "courier new" , "courier" , monospace;"># echo "export PATH=/opt/arcbck/allowed_commands" > .bashrc</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># ln -s .bashrc .profile</span><br />
<br />
The reason for having both a .profile and a .bashrc is to ensure that this profile will be loaded both for interactive and non-interactive sessions.<br />
<br />
If the user needs to write stuff somewhere, create a directory for Bob, e.g.<br />
<span style="font-family: "courier new" , "courier" , monospace;"># mkdir /home/bob/writable</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># chown bob home/bob/writable</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># chmod 755 /home/bob/writable</span><br />
<br />
Create the <i>allowed_commands</i> directory and put symlinks in it pointing to allowed binaries:<br />
<span style="font-family: "courier new" , "courier" , monospace;"># mkdir /home/bob/allowed_commands</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"># ln -s /bin/mycmd allowed_commands/mycmd</span><br />
<br />
Now you must be sure of the following:<br />
<br />
1. Bob must NOT have any writable access to <i>/home/bob/.profile</i> or <i>/home/bob/.bashrc,</i> else he can change the PATH value<br />
2. Bob must NOT have any writable access to <i>/home/bob</i>, to prevent any modification of .profile and .bashrc<br />
3. Investigate ANY command that ends up in the <i>allowed_commands</i> jail to be sure that there is NO known way of executing another command from it, showing files or escaping the shell. If there are any, then forfeit giving this command or write a wrapper around it (see below).<br />
4. See the jail escape methods linked above, log in as Bob and see if you can use them to escape the jail.<br />
<br />
<h4>
Example of a wrapper script with scp</h4>
Let's say I want to allow Bob to scp files into his account using scp's undocumented -t (i.e. -to) option. I would normally do this:<br />
<span style="font-family: "courier new" , "courier" , monospace;"># ln -s /bin/scp allowed_commands/scp</span><br />
<br />
This is <b>wrong</b> as scp can be coerced with -S to execute random commands.<br />
<br />
A solution is to put the following in the <i>allowed_commands </i>jail instead:<br />
<span style="font-family: "courier new" , "courier" , monospace;">lrwxrwxrwx. 1 root root 14 May 5 10:02 scp -> scp_wrapper.sh</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">-rwxr-xr-x. 1 root root 382 May 5 13:54 scp_wrapper.sh</span><br />
<div>
<br /></div>
<div>
With scp_wrapper.sh containing this:</div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">#!/bin/sh</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">if [[ "$1" = "-t" && "$2" != "-"* ]]</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">then</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> /bin/scp -t $2</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> returncode=$?</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">else</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> echo "scp_wrapper: Refused SCP command: '$*'"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> returncode=255</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">fi</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">exit ${returncode}</span></div>
</div>
<div>
<br /></div>
<div>
Using this wrapper, scp will only allow -t and no other option.</div>
<br />
Good luck.Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-73456885775587919882017-04-06T10:05:00.002-07:002017-04-06T18:14:10.847-07:00Applications crash on SLES12 due to lock elisionThis issue has been discussed <a href="http://www.bityard.org/blog/2016/08/05/debugging_segfaults_open-iscsi_iscsiuio_intel_broadwell">in</a> <a href="https://software.intel.com/en-us/forums/intel-isa-extensions/topic/675036">other</a> <a href="https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1642390">places</a>, but mostly related to specific applications and I think it needs its own post here for those who would stumble on this following a Google search.<br />
<br />
<a href="https://sourceware.org/ml/libc-alpha/2013-08/msg00160.html">glibc 2.18</a>, released in 2013, came with a new feature named <a href="https://en.wikipedia.org/wiki/Transactional_Synchronization_Extensions">TSX Lock Elision</a>.<br />
<br />
Briefly, this feature changes the behaviour of libpthread.so in the way it handles mutexes on some specific processors that support hardware lock elision. Intel Xeon CPUs, in particular, support TSX since around 2013 or so. Lock elision offers significant performance gains for some software such as databases.<br />
<br />
You can see if your Linux server's CPU supports lock elision by checking <span style="font-family: "courier new" , "courier" , monospace;">/proc/cpuinfo</span>. If it mentions "hle" (hardware lock elision), it does.<br />
<br />
RHEL 7 does not support this as of now. It comes with glibc 2.17, so lock elision is not enabled on these systems. As for SLES12, it comes with glibc 2.19, which means that SLES12 systems will use lock elision if the CPU supports it.<br />
<br />
However, if an application unlocks a mutex twice, this can cause problems if lock elision is enabled. This is explained in detail in an <a href="https://lwn.net/Articles/534758/">LWN article</a>. Let me quote an important paragraph in this article:<br />
<i><br /></i>
<br />
<blockquote class="tr_bq">
<i><tt style="background-color: white;">pthread_mutex_unlock()</tt><span style="background-color: white;"> detects whether the current lock is executed transactionally by checking if the lock is free. If it is free it commits the transaction, otherwise the lock is unlocked normally. This implies that if a broken program unlocks a free lock, it may attempt to commit outside a transaction, an error which causes a fault in RTM. In POSIX, unlocking a free lock is undefined (so any behavior, including starting World War 3 is acceptable). It is possible to detect this situation by adding an additional check in the unlock path. The current glibc implementation does not do this, but if this programming mistake is common, the implementation may add this check in the future.</span></i></blockquote>
<br />
<br />
The "programming mistake" here is double-unlocking mutexes. I've made a sample C program that does exactly this, and although it works fine with glibc 2.17, it will crash on glibc 2.19 with a segmentation fault in <span style="font-family: "calibri" , sans-serif; font-size: 11pt;">__</span><span style="font-family: "courier new"; font-size: 10pt;">lll_unlock_elision()</span><span style="font-family: inherit;">, if, and only if, the server's cpuinfo reports "hle".</span><br />
<span style="font-size: 10pt;"><span style="font-family: inherit;"><br /></span></span>
I've stumbled upon a few applications, which I will not name here, that crash on SLES12. Upon analyzing their cores, I found that they have this same exact problem with <span style="font-family: "calibri" , sans-serif; font-size: 11pt;">__</span><span style="font-family: "courier new"; font-size: 10pt;">lll_unlock_elision()</span>. So, one can assume that they might double-unlock some mutexes.<br />
<br />
The bottom line is that if you have an app that does this, your best bet is to contact the vendor, and ask them to remove double mutex unlocks in their code, if they have any.<br />
<br />
If that is not possible, there are two workarounds:<br />
<br />
1. The first is to patch /etc/ld.so.conf to override libpthread 2.19 with a version that is compiled with lock elision disabled. <a href="https://www.novell.com/support/kb/doc.php?id=7017650">This is documented in Novell's KB here</a>.<br />
<br />
2. The second (and preferred) solution is to adjust LD_LIBRARY_PATH to override it on a per-application basis. You could therefore change its startup script to add this:<br />
<br />
<div class="MsoNormal">
<span lang="EN-CA" style="font-family: "courier new"; mso-ansi-language: EN-CA;">LD_LIBRARY_PATH=/lib64/noelision:/lib/noelision:$LD_LIBRARY_PATH</span></div>
<div class="MsoNormal">
<span style="font-family: "courier new";">export
LD_LIBRARY_PATH</span></div>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Hope this helps.</span><br />
<span style="font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"></span><span style="font-family: "calibri" , "sans-serif"; font-size: 11.0pt;"></span>Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-72497041631648766902016-09-02T07:18:00.003-07:002016-09-02T12:35:49.810-07:00Sending text logfiles from Windows to a syslog server, reliablyI'm in the following situation:<br />
<ol>
<li>I have a Windows application, let's name it <i>MyApp</i>. </li>
<li><i>MyApp </i>creates important log files on my server without using the Event Log. These log files are simply textfiles (i.e. logfile.txt)</li>
<li>For compliance purposes, I have to send these log files to a remote syslog server.</li>
<li>The compliance auditor wants me to ensure that these log files are <b>always</b> sent no matter what.</li>
</ol>
It doesn't matter what the application is (as long as it creates a text file somewhere) and wether the receiving end is an Arcsight appliance, a Splunk box, or syslog-ng: This post will describe a generic way to achieve this, with the added bonus of reliability.<br />
<br />
The two products that I used to implement this are <a href="http://sourceforge.net/projects/neologger/">neologger</a> and <a href="http://nssm.cc/">NSSM</a>:<br />
<ul>
<li><b>Neologger</b> reads a file (as a mater of fact, it tails it) and sends it to a syslog server. </li>
<li><b>NSSM</b> is a software that lets you wrap any application (in our case, Neologger) in a Windows service. </li>
</ul>
<blockquote class="tr_bq">
<span style="font-family: "trebuchet ms" , sans-serif; font-size: x-small;"><b>What is "tailing" a file?</b><br />Unix administrators are familiar with the <i>tail</i> command: it follows a text file, grabbing new entries at the end as they come in. Neologger, basically replicates what "<span style="font-family: "courier new" , "courier" , monospace;">tail file.log | logger</span>" would do on Unix.</span></blockquote>
<br />
<h3>
Using Neologger</h3>
Neologger is, in essence, a simple and reliable tool. It will tail a text file endlessly, and it automatically detects if that file is deleted, shrunk or rotated, which ensures a reliable operation. To use it, simply try:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"># <b>neolog.exe -r logfile.txt -tail -t <i>syslog_server</i> -d</b></span><br />
<br />
This will tail file <i>logfile.txt</i> and send it to <i>syslog_server</i>. Many other command-line options are available. Note the <b>-d</b> option; this is a debug option that lets you see what it does, you normally would not want it there.<br />
<br />
The first thing you need to do is therefore to craft a command-line as above, but specific for your application. Here is a more complete example:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># <b>"C:\Program Files\neolog\neolog.exe" -r "C:\ProgramData\My App\logfile.txt" -tail -t mysyslogserver.company.com -p 1234 -d</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
This will tail <i>logfile.txt</i> and send it to <i>mysyslogserver.company.com</i> on port 1234. Once it works for you, remove the <b>-d</b> option.<br />
<br />
<br />
<h3>
Wrapping Neologger with NSSM</h3>
Now, the next question is, <i>how do I ensure that neolog.exe runs reliably</i>? The answer is to configure Neologger as a service under Windows. It's easier to manage as a service and the operating system will ensure that it restarts appropriately if it ever crashes. That's where NSSM comes into play. NSSM (Non-Sucking Service Manager) is a tool that lets you wrap almost <i>any</i> application as a service.<br />
<div>
<br /></div>
To create a service to wrap Neologger, run NSSM like this:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># <b>nssm install MyApp-Syslog</b></span><br />
<br />
This will create a new service named <i>MyApp-Syslog</i>. Then, fill the <i>Path</i>, <i>Startup directory</i>, and <i>Arguments</i> as appropriate (don't forget to remove <b>-d</b> as it is not required here). Here is an example:<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhfBImZnfMSFSiBRTEXPJy1coTLITF8gPWAXhHD8wO-NBVwMvYZIoLuI-6y42YUXS7EzqIthk7tRd-asVTwUKaMgdzYk_F2Tjrov2NUMr4iQX1ph7Kw1614CJDKH017HSEbrMwHl-Qbc8/s1600/Capture.PNG" imageanchor="1" style="font-family: "Courier New", Courier, monospace;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhfBImZnfMSFSiBRTEXPJy1coTLITF8gPWAXhHD8wO-NBVwMvYZIoLuI-6y42YUXS7EzqIthk7tRd-asVTwUKaMgdzYk_F2Tjrov2NUMr4iQX1ph7Kw1614CJDKH017HSEbrMwHl-Qbc8/s1600/Capture.PNG" /></a><br />
<br />
You don't need to change anything in the other tabs, but you can take a look in case you need to fine-tune something.<br />
<br />
Now you can try starting the <i>MyApp-Syslog</i> via the service panel, and see if it works.<br />
<blockquote class="tr_bq">
<span style="font-family: "trebuchet ms" , sans-serif; font-size: x-small;"><b>What happens if the log file isn't there in the first place? </b>While neologger will "wait" if the file disappears once it starts tailing it, it will gracefully exit if it's not initially there. NSSM will then try to restart neolog.exe <a href="https://nssm.cc/usage#throttling">using its throttling settings</a>. This ensures that the service will loop neolog.exe, slowly, until the file appears again. During that time, the service is labeled as "Paused" in the service panel.</span></blockquote>
<br />
<h3>
Going a step further with dependencies</h3>
<div>
The last step, which can be important for compliance reasons, is not only to help Neologger run reliably (which is done by configuring it as a service), but ensure that it <i>always</i> runs when your application runs, too. This is done with dependencies.</div>
<div>
<br /></div>
<div>
If your application doesn't run as a service, you're out of luck. But let's say <i>MyApp</i> runs under a Windows Service named <i>MyApp-Service</i>. It then becomes trivial to make <i>MyApp-Service</i> depend on <i>MyApp-Syslog. </i></div>
<div>
<i><br /></i></div>
<div>
To change dependencies, you have to edit <i>MyApp-Service</i> directly. First, query <i>MyApp-Service</i> to see if it has other dependencies:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"># <b>sc qc MyApp-Service</b></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[SC] QueryServiceConfig SUCCESS</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SERVICE_NAME: MyApp-Service</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> TYPE : 10 WIN32_OWN_PROCESS</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> START_TYPE : 2 AUTO_START</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ERROR_CONTROL : 1 NORMAL</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> BINARY_PATH_NAME : "C:\Program Files\MyApp\MyApp.exe"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> LOAD_ORDER_GROUP :</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> TAG : 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> DISPLAY_NAME : MyApp Service</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> DEPENDENCIES : tcpip</span></div>
<div>
<br /></div>
</div>
<div>
You can see here that MyApp-Service depends on <i>tcpip</i>. It is important to keep this in mind. Next, change the dependencies on <i>MyApp-Service</i> by configuring it to depend on both <i>tcpip</i> and <i>MyApp-Syslog. </i>Note here that you have to explicitly state that<i> tcpip </i>is still a dependency, and separate it with a slash to add <i>MyApp-Syslog.</i></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"># <b>sc config MyApp-Service depend= tcpip/MyApp-Syslog</b></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[SC] ChangeServiceConfig SUCCESS</span></div>
</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"># <b>sc qc MyApp-Service</b></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">[SC] QueryServiceConfig SUCCESS</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">SERVICE_NAME: MyApp-Service</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> TYPE : 10 WIN32_OWN_PROCESS</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> START_TYPE : 2 AUTO_START</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> ERROR_CONTROL : 1 NORMAL</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> BINARY_PATH_NAME : "C:\Program Files\MyApp\MyApp.exe"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> LOAD_ORDER_GROUP :</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> TAG : 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> DISPLAY_NAME : MyApp Service</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> DEPENDENCIES : tcpip</span></div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> MyApp-Syslog</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<br /></div>
<div>
Once this is done, start <i>MyApp-Service</i>. You'll notice that it starts <i>MyApp-Syslog</i> automatically. The same logic applies if you stop <i>MyApp-Syslog</i> before <i>MyApp-Service</i>, both will stop at the same time.</div>
<div>
<br /></div>
<div>
<h3>
Putting it all together</h3>
</div>
<div>
To conclude, let's restate what we just did. First, we used Neologger to tail a text file on Windows, generated by an application named <i>MyApp</i> and sent it, live, to a syslog server. Then, we used NSSM to configure Neologger as a Windows service to help us manage its startup and shutdown. Finally, we created a dependency between the service that runs <i>MyApp</i> and the new service we've just created, to reassure our compliance auditor that Neologger <b>always</b> runs when <i>MyApp</i> runs, too.</div>
<div>
<br /></div>
<div>
Good luck.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<i><br /></i></div>
<div>
<i><br /></i></div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-12259938078448969712016-08-23T11:49:00.002-07:002016-08-23T11:50:52.455-07:00Running "MRPE" check_mk scripts asynchronously on WindowsI have a corner case on Windows where I need to execute classic Nagios NRPE scripts within check_mk, but in asynchronous mode. These scripts can, in certain circumstances such as a network timeout, take a significant time to execute and they cannot be run from the check_mk agent.<br />
<br />
It's possible to have honest-to-goodness check_mk scripts execute asynchronously, using the <span style="font-family: "courier new" , "courier" , monospace;">async</span> directive in check_mk.ini. I tried it, it works. However, this is not supported by the agent with classic nagios plugins.<br />
<br />
So, I wrote a wrapper named <b>mrpe_async_wrapper </b>that does just that. It's not rocket science; the wrapper is simply a Windows batch file that:<br />
<br />
<ol>
<li>Creates a scheduled task (on its first run) that executes the check script at 5 minutes intervals;</li>
<li>The scheduled taks instructs mrpe_async_wrapper to run the check script and save its output in a status file;</li>
<li>When run directly, mrpe_async_wrapper reports the contents of the status file instead of executing the script. It does it <i>quickly</i>. So, you can run it each minute if you want, but it will only report the status within up to the last 5 minutes. </li>
</ol>
<br />
This lets you run slow or unpredictable NRPE scripts from check_mk without fear. I've been running this for a few days and it seems to do the job for me.<br />
<br />
To configure it, simply add a directive to the [mrpe] section of check_mk.ini like this (on the same line)<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">check = check_gizmo C:\tools\mrpe_async_wrapper.bat check_gizmo C:\tools\check_gizmo.bat</span><br />
<div>
<br />
<br />
This defines an MRPE check named "check_gizmo", which instructs the wrapper to create a scheduled task named "check_gizmo" that runs c:\tools\check_gizmo.bat asynchrnously.<br />
<br />
Here is the code for the wrapper:<br />
<a href="https://drive.google.com/open?id=0B_-YLkocOctyTW43dFlkeHZlWnM">mrpe_async_wrapper.bat</a><br />
<br />
Have fun.</div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com2tag:blogger.com,1999:blog-1718921294435140810.post-62641622347635770842016-06-27T12:50:00.001-07:002016-06-27T13:01:11.158-07:00Getting UFO2 failover status from an OSIsoft PI Interface<h4>
Overview</h4>
I'm currently deploying an OSIsoft PI Interface node at my workplace.<br />
<br />
Being a "Systems" Administrator, and not a "PI" Administrator per se, I was looking for a way to get high-availability status directly from that interface node. My objective was to provide IT Operations with an easy-to-use procedure that answers the following question: <b>Which interface node is currently active and which one is currently in standby?</b>... It is useful for them to know the answer to this when scheduling maintenance such as Windows patches.<br />
<br />
Unfortunately, there is no <i>easy</i> way to find out which of the two interfaces is currently active. I've looked everywhere in OSIsoft's KB and I guess nobody asked. :-)<br />
<div>
<b><br /></b></div>
<div>
<b><br /></b></div>
<div>
<b>Some information on UFO </b></div>
<br />
Many, if not all, PI interfaces are based on UniInt (Universal Interface). UniInt supports two failover levels named UFO (UniInt FailOver):<br />
<br />
<ul>
<li>UFO phase 1 (UFO1) which is based on PI points</li>
<li>UFO phase 2 (UFO2) which uses a shared file located on a separate file server</li>
</ul>
<br />
Not all interfaces support UniInt failover; check your Interface documentation. Mine only supports UFO2.<br />
<br />
You can look at the following KBs for more information:<br />
<br />
<ul>
<li>UniInt phase 2 Failover scenarios: <a href="https://techsupport.osisoft.com/Troubleshooting/KB/KB01346">https://techsupport.osisoft.com/Troubleshooting/KB/KB01346</a></li>
<li>Comparing Phase 1 and Phase 2 Failover of the PI OPC Interface: <a href="https://techsupport.osisoft.com/Troubleshooting/KB/KB00446">https://techsupport.osisoft.com/Troubleshooting/KB/KB00446</a></li>
</ul>
<br />
UFO2 is preferred to UFO1, and KB00446 even mentions that UFO1 is deprecated. That might be due to the fact that I see one major drawback with UFO1: if one node looses access to the PI Server, it cannot know the status of the other node. Using a shared file on a file server (a highly-available one, that is!) is deemed more reliable.<br />
<div>
<br /></div>
<h4>
Finding what interface is active, the PI Admin Way</h4>
There seems to be one official way, the "PI Admin Way", which involves looking up points stored in the PI Server.<br />
<br />
While my interface is UFO2, it seems to create PI points anyway. These points are created directly from ICU, and they all have "UFO2" in their names. It is therefore trivial to check their values from the PI SDK Utility tool. For example:<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6kn6WkNU48BDUWj5Ky0BJ4ZoKfcIrqKD7clmacCW_xYjhsdWOY6fd08PBSR6HboS6hn-OzkP6-A5ThB5rb78SPPj8xg6vzbd4zMuMM9wug1zpFpVFUXdcZWrA9UGKFeTESGRCI8Usa10/s1600/1.png" imageanchor="1"><img border="0" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6kn6WkNU48BDUWj5Ky0BJ4ZoKfcIrqKD7clmacCW_xYjhsdWOY6fd08PBSR6HboS6hn-OzkP6-A5ThB5rb78SPPj8xg6vzbd4zMuMM9wug1zpFpVFUXdcZWrA9UGKFeTESGRCI8Usa10/s640/1.png" width="640" /></a><br />
<br />
PRO TIP: It's also possible to find out these values at the command line using <i>apisnap</i>.<br />
<br />
<div>
While this is sufficient from a PI admin perspective, from a systems administrator perspective, it's not great. For instance, it's not an easy task for IT Operations to fire up that tool and query PI points, it cannot be automated in a script (except if using apisnap) and lastly it will not work at all if the nodes cannot speak to the PI server. It is thus preferable to ask them to run a simple command.</div>
<div>
<br /></div>
<h4>
Finding what interface is active, the born-again Sysadmin Way</h4>
<div>
It was a simple task to somewhat reverse-engineer the binary UFO2 .dat file created by the interface and write a simple program to extract basic data. I've named it <i>readdat</i>.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">C:\tools><b>readdat \\myfileserver\myfile.dat</b></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Active Node (0 = None, 1 = Node 1 is primary, 2 = Node 2 is primary)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Active ID: 1</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Device Status (0 = Good, 99 = OFF, any value in between results in a failover)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Node 1: 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Node 2: 0</span></div>
</div>
<div>
<br /></div>
<div>
Works good enough for me. Readdat.exe can then wrapped in a batch file or a powershell script to make it easier to use.</div>
<div>
<br /></div>
<div>
As a bonus, you can run it like this:</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">C:\tools></span><b style="font-family: "Courier New", Courier, monospace;">readdat \\myfileserver\myfile.dat -activeid</b></div>
<div>
<br /></div>
<div>
This will set ERRORLEVEL to the ID number.</div>
<div>
<br /></div>
<div>
The source code for readdat is here:</div>
<div>
<a href="https://drive.google.com/open?id=0B_-YLkocOctyUUxyZFRnY2dia1U">readdat.c</a></div>
<div>
<br /></div>
<div>
Here is also a Win32 executable:</div>
<div>
<a href="https://drive.google.com/open?id=0B_-YLkocOctyX002MHZqYUFOVFE">readdat.exe</a></div>
<div>
<br /></div>
<div>
Good luck!</div>
<div>
<br /></div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-6038834776489739562015-09-14T11:14:00.000-07:002016-08-23T11:57:35.231-07:00Configuring vsftpd to support proxy FTP<div class="MsoNormal">
<br />
I've had to deal with a legacy application that is hard coded to use proxy ftp sessions. These are initiated by using the "proxy" command in a stock ftp client. <br />
<br />
It was giving us trouble with vsftpd refusing to transfer files when using "proxy get" to initiate a passive session between the vsftpd server and another server.<br />
<br />
What is a proxy FTP? In a nutshell, a proxy session lets you open a connection to a second FTP server, so that you can transfer files between both servers from instead of between the primary server and your FTP client.<br />
<br />
The ftp(1) man page documents what "proxy" does. It is important to read it and understand what happens when you use this:<br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>proxy</b> <u>ftp-command</u></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Execute an ftp command on a secondary control connection.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> This command allows simultaneous connection to two remote ftp</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> servers for transferring files between the two servers. The</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> first <b>proxy</b> command should be an <b>open</b>, to establish the sec-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ondary control connection. Enter the command "proxy ?" to</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> see other ftp commands executable on the secondary connec-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> tion. The following commands behave differently when pref-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> aced by <b>proxy</b>: <b>open</b> will not define new macros during the</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> auto-login process, <b>close</b> will not erase existing macro defi-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> nitions, <b>get</b> and <b>mget</b> transfer files from the host on the</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> primary control connection to the host on the secondary con-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> trol connection, and <b>put</b>, <b>mput</b>, and <b>append</b> transfer files</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> from the host on the secondary control connection to the host</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> on the primary control connection. Third party file trans-</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> fers depend upon support of the ftp protocol PASV command by</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> the server on the secondary control connection.</span><br />
<div>
<br /></div>
So how does this impact vsftpd when using it to handle the primary control connection?<br />
<br />
The first thing that might happen is that if you issue a proxy get, it might fail with the following message:<br />
<b><span style="font-family: "courier new" , "courier" , monospace;">500 Illegal PORT command</span></b><br />
<b><br /></b>
This is fixed by adding the following parameter to vsftpd.conf:</div>
<div class="MsoNormal">
<span style="font-family: "courier new";">port_promiscuous=YES</span><br />
What this parameter does is authroize vsftpd to open a data connection with the proxy server, instead of limiting it between vsftpd and the FTP client.</div>
<br />
Then, you might get:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><b>500 OOPS: vsf_sysutil_bind</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b><br /></b></span>
This happens because the vsftpd process is trying to bind to port 20 to the IP address of the server. By stracing the process, I found out that this does not work because the vsftpd process that handles communication with clients is unprivileged. This privilege separation is by design. The workaround I found is to add this to vsftpd.conf:<br />
<span style="font-family: "courier new";">connect_from_port_20=NO</span><br />
<div class="MsoNormal">
<br />
This makes vsftpd bind to another port (I didn't even check which one) but it works. By default it is set to "NO", but it is left to "YES" in the example configuration file and thus why it was there in the first place.<br />
<br />
Good luck</div>
Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com1tag:blogger.com,1999:blog-1718921294435140810.post-90447081734209406202015-07-20T20:24:00.002-07:002015-07-24T18:24:24.158-07:00Updating a Magellan Triton 500's GPS chip firmwareI've been recently trying to restart using a circa-2008 GPS I own, a <a href="http://www.magellangps.com/Store/Triton/Magellan-Triton-500-North-America">Magellan Triton 500</a>. Back when I purchased it, it was so frustrating to use that I gave up. It's time for a rematch.<br />
<br />
There used to be a <a href="http://www.tritonforum.com/forum/">english forum</a> with lots of information on these, but it closed some years ago. Whatever was in this forum is lost forever (and no, the web archive didn't save the posts, only the thread subjects).<br />
<br />
There are still tidbits of info scattered here and there, however. Many in russian and german though, which requires running them through a translator, with mixed results. I might try to put a comprehensive page in the future in this blog in case they go down, too.<br />
<br />
In the mean time, here is the best hack available for this unit. I found an interesting post in a German board that explains how to flash an <i>unofficial</i> driver for the <a href="https://en.wikipedia.org/wiki/SiRFstarIII">SiRFstar III</a> GPS chip that updates its software from GSW 3.2.4 to 3.5.0 and it<a href="https://www.youtube.com/watch?v=MlplCEpB7cM"> increases the unit's sensitivity considerably</a>. The details are here:<br />
http://www.magellanboard.de/viewtopic.php?f=35&t=4343<br />
<br />
Here is how to do this.<br />
<br />
1. Download this file here:<br />
http://maps4me.net/tools/Triton500_UPD1.95_SIRF.zip<br />
P.S. maps4me offers many maps for the triton for a one-time download fee, I suggest you check it out.<br />
<br />
2. Extract the zip file.<br />
<br />
3. Turn OFF your GPS (this is important, if it's already turned on when you plug it in, the GPS driver will fail to install)<br />
<br />
4. Run MgnFwUpd.exe<br />
<br />
5. NOW plug your GPS, turn it on, and run the update. It takes at least 30 minutes to complete.<br />
<br />
6. Tada! The about -> version page should show GSW3.5.0 for SiRF.<br />
<br />
I've tested this on Windows 8.1 and it still works even if the software was probably designed for XP.<br />
<br />
For advanced users: If you don't want SiRF 3.5.0, there is a way to update from 3.2.4 to 3.2.5 using official Magellan code. I requires downloading the latest firmware update to 1.95 from Magellan, running their update, finding out the temporary directory where it extracts its data, then modifying MgnFWUpd.xml to uncomment the line mgnFWGpsChipUpdate version. I tried it but didn't find 3.2.5 to be very useful.Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com1tag:blogger.com,1999:blog-1718921294435140810.post-90293301236046289392014-09-22T17:52:00.004-07:002014-09-22T17:54:29.760-07:00Change control killed the sysadmin star<br />
Yesterday, I've watched Office Space<i> </i>for the first time in probably 10 years. I can't believe how this movie is still relevant today. What's most funny is that one of the center pieces of this movie, <a href="http://en.wikipedia.org/wiki/TPS_report">TPS reports</a>, is the type of report I have to file sometimes. Even if the movie's underlying themes have not aged much, there is one thing that Mike Judge would have to consider if he had to direct a reboot (pun intended) of Office Space 2015: change control.<br />
<br />
I could go on and on about change control but I won't. However, I can leave you with this song:<br />
<br />
<i>Red tape came and broke your heart</i><br />
<i>We can't approve you've gone too far</i><br />
<i>Change control killed the sysadmin star</i><br />
<div>
<i><br /></i></div>
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com1tag:blogger.com,1999:blog-1718921294435140810.post-52000150300598417822013-10-03T17:13:00.001-07:002013-10-03T17:13:59.645-07:00Running a Matrox G450x4 MMS under Windows 7 <div class="separator" style="clear: both; text-align: center;">
<a href="http://www.matrox.com/graphics/media/image/products/graphics_cards/mms_series/g450mms4.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="228" src="http://www.matrox.com/graphics/media/image/products/graphics_cards/mms_series/g450mms4.jpg" width="320" /></a></div>
<br />
I was recently tasked with a small challenge. Given that we have a fair amount of circa-2005, quad-screen workstations running Windows XP for which we know <a href="http://windows.microsoft.com/en-CA/windows/end-support-help">the clock is ticking</a>, is it possible to upgrade them to Windows 7 even if they have ancient Matrox graphic cards?<br />
<br />
The answer is, yes, with some limitations. Matrox doesn't have a clear stance on Win7 support for the G450 series. By downloading their latest driver which is supposed to support Win7 SP1, the installer fails without even a hint of what is going on.<br />
<br />
By searching for and trying various older drivers, I found out that the WHQL drivers do not support the G450, but the non-WHQL do. To get these drivers, you have to go to the "<a href="http://www.matrox.com/graphics/en/support/drivers/previous/">archived support drivers</a>" area and scroll down to the latest non-WHQL driver you can find for your platform. In my case, it was version <a href="http://www.matrox.com/graphics/en/support/drivers/download/?id=476">211_00_183</a>. The driver installs and the graphic card works. Case closed.<br />
<br />
Of course, by using non-WHQL drivers, you might be asked by Microsoft to remove these drivers if you run into problems and ask for support.<br />
<br />
For the curious, these workstations have been limited to being quad-screen ICA clients a long time ago, so I don't expect any performance impact by moving them to Windows 7. If we move on with this scenario, we'll be saving the company some money by extending the life of this equipment for a few more years.<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com6tag:blogger.com,1999:blog-1718921294435140810.post-72561532954606549722013-05-08T07:06:00.002-07:002013-05-08T07:06:48.635-07:00Launching the Performance Monitor from the command line or script<br />
On a Windows 2008 R2 server, I needed to launch the Performance Monitor with a built-in live report. That is targeted to support personnel and I don't want them to have to start it an add counters manually each time.<br />
<br />
Man, that task proved to be more complex than I expected.<br />
<br />
Here is what I've found in the last few days:<br />
<br />
<b>Solution 1: use IE</b><br />
From the performance manager, the only option offered to save a custom report is to save it to an .HTML file which can then be launched from IE. That is clunky, as it has some ActiveX code and you need to acknowledge running it. Furthermore, when you load up that HTML page, you first have to press on the "play" icon to start the data collection, which is another useless step that I don't want support guys to have to do.<br />
<br />
<b>Solution 2: use Typeperf</b><br />
There is a nice utility named "typeperf.exe" that can be used to dump specific counters to the console. It works, but for an odd reason, it can ONLY output CSV output to the screen. If you specify another format, it insists on dumping in a file. In essence it is a good quick-and-dirty tool for the console but not a terrific all-around solution.<br />
<br />
<b>Solution 3: use Perfmon in standalone mode (WE HAVE A WINNER!)</b><br />
You can launch a standalone Permon using "perfmon /sys". This lets you add counters and, look at the magic, the standalone panel offers the possibility of saving that report in a .<i>PerfmonCfg</i> file. To load the file, simply click on it (or use "start meh.PerfMonCfg" within a batch file) and it will bring up a good old Perfmon report on the screen. That, in my opinion, is the best way to achieve my goal.<br />
<br />
Sorry for the lack of details, but that should give you an idea.<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-68060414396972116532012-06-04T17:53:00.002-07:002012-06-04T17:53:38.318-07:00The born-again sysadminA few weeks ago, I decided to go back to my roots as a systems administrator. While being an architect was a fulfilling experience, I was missing a lot the technical work I used to do in my previous career.<br />
<br />
So, I'll be transitioning from my current architecture duties to join a team in charge of a mission critical system that currently runs on - hold your breath - <a href="http://en.wikipedia.org/wiki/Tru64_UNIX">Tru64 Unix</a>. This is a major real-time system for my company, which spans three sites and counts a plethora of servers. Can't say more due to security issues. Many in that team are due to leave for retirement this summer so I'll probably work as a <a href="http://www.indiewire.com/static/dims4/INDIEWIRE/8c0b498/4102462740/thumbnail/680x478/http://d1oi7t5trwfj5d.cloudfront.net/dd/e90150736d11e1bcc4123138165f92/file/Mr_wolf.jpeg">Mr Wolf</a> for a while, fixing what needs to be fixed (and strategically avoiding what doesn't need to).<br />
<br />
This Tru64 system is planned to be upgraded to Linux (specifically RHEL 5.8) starting next year, with a go live in 2014. And I'm in the delightful situation of having been one of the architects to document that upgrade -- the difference being that it's ME who will have to live with it for years to come.<br />
<br />
I'll be the first to admit that this blog has been slow going between 2010 and 2012. There was simply no content that was "generic" enough to be published here. Things might change over time.<br />
<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-79563858286773510202012-04-16T07:52:00.004-07:002012-04-16T07:56:14.438-07:00The Microsoft SMTP service doesn't create a log file.The Microsoft SMTP service doesn't log anything in the SmtpSvc1 directory even though you enabled logging and you're on Windows 2003?<div><br /></div><div>Save yourself some trouble, and install the "ODBC Logging" role service in Server Manager. Although it shouldn't help, it does, and for me the SMTP service started logging automagically after restarting it.</div><div><br /></div><div>Wasted an hour on that nonsense before finding this trick buried deep somewhere in a forum posting.</div><div><br /></div><div><br /></div><div><br /></div><div><br /></div>Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-2551115234166232452011-11-04T08:07:00.001-07:002011-11-04T08:20:05.955-07:00A homebrew ATSC multi-room PVR project<div>I'm not involved in technical matters much at work right now and thus I've fallen back to updating my own home on my spare time. This is a six-month project that will consist of:</div><div><br /></div><div>Canada has switched to ATSC on September 1st, so my main objective is to cut cable TV and modernize my 2 current standalone cable-company-supported-PVRs which have an interface that dates back to 2001.</div><div><br /></div><div>In a nutshell, I'll do this in the following weeks:</div><div><ul><li><b>No more lightning in the house</b>: Installing and grounding an exterior OTA antenna</li><li><b>Thank you foxconn and the MediaPortal Team</b>: Building and configuring a "budget" Windows 7 TV Server with MediaPortal</li><li><b>Rsync now, robocopy later</b>: Dismantling my FreeNAS-based NAS to consolidate data on the TV Server.</li><li><b>PXE for the masses</b>: Deploying 2-3 MediaPortal client nettops using PXE and OpenWRT</li></ul></div><div><br /></div><div>I've got some of the pieces in place. The Win7 server is running and I should be ready to test MediaPortal soon. Why Win7 and not Win2008? The reason is that my ATSC card (an <a href="http://www.avermedia-usa.com/AVerTV/product/ProductDetail.aspx?Id=474">AverTVHD Duet</a>) doesn't have drivers for 2008, and I don't need a domain controller for my house anyway. For the nettops, I have one in hand already, and wish to try installing them using PXE (just for the kicks).</div><div><br /></div><div>I'll try to post some pictures and details over the coming weeks. They should roughly follow the 4 steps above.</div><div><br /></div><div>Olivier</div><div><br /></div>Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-82063343681311494102011-09-27T18:03:00.000-07:002011-09-27T18:22:18.609-07:00Yes, I'm still alive.Not many posts lately huh?<br /><br />My older blog, <span style="font-style: italic;">Technocrat-UX</span>, consisted of a way for me to document quirks and techniques related to HP-UX, BladeSystems, and some other technologies. <span style="font-style: italic;">Technocrat-UX</span> enjoyed some success as these were niche, but relevant, subjects.<br /><br />That model doesn't fit well with <span style="font-style: italic;">The ex-sysadmin</span> where I originally had the intention of documenting my new job as a systems architect. The main problem when designing IT architectures is that the ideas and diagrams that result of my efforts are not generic and reusable enough, thus not interesting. Furthermore, in a security perspective, a lot of work needs to be done to obfuscate the information - any information - before it is released. I can't, for instance, publish a networking topology just like that to the public.<br /><br />Up until recently I did, however, have the intention of writing a paper and presentation documenting a reference architecture for IED event and measure collection following my 18 month experience with Cooper's products. But due to some restrictions, that has not been possible yet.<br /><br />In the mean time I'm keeping the blog going with posts that I *think* could be interesting to sysadmins, architects and... ex-sysadmins.<br /><br />O.Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-43415795278552736702011-08-30T06:04:00.000-07:002011-08-30T06:07:04.442-07:00HP's Power AdvisorThis morning I had to use HP's Power Advisor to estimate the load of small servers I need to deploy (DL360G7s). I remember using an older tool some years ago but this new one is much better. It's available here:
<br /><a href="http://h18004.www1.hp.com/products/solutions/power/advisor-online/HPPowerAdvisor.html">http://h18004.www1.hp.com/products/solutions/power/advisor-online/HPPowerAdvisor.html</a>
<br />
<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqm5_nZyhrFEEUzsdvtIiG07jafdjv9ZEjQVm3OR509PzXdwG4NhPZFReXdxREglqVefLm8EbOwk5fa8_Mn2f7HjBG8k5P7Ij-fdpQkVCY6T2y6nN5vnqrHD-r4N336PuKmdu8f70qgRk/s1600/3.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 320px; height: 220px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqm5_nZyhrFEEUzsdvtIiG07jafdjv9ZEjQVm3OR509PzXdwG4NhPZFReXdxREglqVefLm8EbOwk5fa8_Mn2f7HjBG8k5P7Ij-fdpQkVCY6T2y6nN5vnqrHD-r4N336PuKmdu8f70qgRk/s320/3.png" alt="" id="BLOGGER_PHOTO_ID_5646634761104364754" border="0" /></a>
<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-87512757542748450362011-08-26T11:41:00.000-07:002011-08-26T12:46:06.132-07:00Dealing with MFT servers: a systems architect versus systems administrator love storyIn my past life as a system administrator, I once had to build from a ground up a secure MFT (<a href="http://en.wikipedia.org/wiki/Managed_file_transfer">managed file transfer</a>) server. I've pulled it off by using the HP-UX infrastructure I was comfortable with, and built something from the ground up using OpenSSH. You wouldn't believe, however, how much tweaking had to be done to have the user accounts (which were stored in /etc/passwd and /etc/shadow) synchronized reliabily in a clustered system spanning two sites. It took me a few days to make sure everything was correct.
<br />
<br />Now it is time to do it again at a new place. I have to design another highly-available MFT server that will be wedged between two DMZs, and besides supporting SFTP I want it to have an HTTPS-based, "drop box" feature for end-users do be able to upload files easily without needing an SFTP client. Oh, and by the way, I need it to be able to authenticate users with a Windows domain this time.
<br />
<br />If I was still a sysadmin, I'd have to extend the first solution further by adding an Apache HTTPD server and some open-source file upload solution. Then, I'd have to find a solution *BOTH* for Apache and OpenSSH to authenticate users. OpenSSH would probably need to rely on PAM, and for Apache I don't have a clue. Yet no problema; I would just shrug and say <span style="font-style: italic;">I can do that</span>, then spend a few days tying everything up. The end.
<br />
<br />But, as a system architect, things don't work this way.
<br />
<br />Why? Because I have to assume that there is no guarantee the sysadmin who will have to do the grunt work of building this up will be willing, or have enough experience, to install and configure a custom solution. And assuming he/she is willing to do it, I have to consider that each man-hour counts for serious dough within the frame of a project. With custom hacks like this, these can sum up to a lot of hours depending on whose desk the work falls on.
<br />
<br />Therefore, I did what system architects do: I tried to pick a turnkey solution, and it will have to be shoved down the IT team's throat.
<br />
<br />I always <span style="font-style: italic;">hated</span> this when it happened before. Picture this: The architect goes on a golf course or whatever, and randomly picks a solution based on bullet points and checklist tables. Then the IT operations guy has to take whatever <a href="http://omasse.blogspot.com/2009/03/ridiculous-price-of-hp-enterprise.html">crappy, slow and expensive "enterprise" software</a> the architect purchased at a pharaonic price, and make it work satisfactorily to fulfill a business need. More often than not, such lame software ends up in the garbage bin with the IT team developing its own in-house solution to patch things up.
<br />
<br />As I've been on that side of the fence before, I try to do things differently to prevent this from happening. So when I technically can, I actually <span style="font-weight: bold;">try out</span> software before choosing it, when it's not too daunting for me to do so.
<br />
<br />So back to the MFT. I went searching on the web and picked a market "enterprise" leader to try it out. Not only doesn't it support high availability easily, the software is clumsy and it took 30 minutes to run its installshield sequence on a Windows 2008 VM. Uninstalling took the same. And to have SFTP support, I actually had to pay a premium over the base price. Not good.
<br />
<br />Few other vendors had solutions that seemed serious enough, though. By serious I mean that they have to offer technical support, and have some agility in dealing with enteprise customers. Then a colleague of mine found out a very elegant software named <a href="http://www.jscape.com/">JSCAPE MFT Server</a>. Installation is a snap and it's very easy to configure. I was up and running in a few minutes. And as a bonus, its feature set is actually useful and seems to have been designed based on user requests instead of some odd crystal ball. I've been trying it this morning and up to now, it works very well.
<br />
<br />The MFT server itself works on Windows, Linux, some Unices and Mac OS X. Installing the RPM went without any problem on CentOS 6. It is managed by a Java-based GUI that I installed on Windows -- I wasn't fond of using a thick-client when compared to a web-based administration GUI, but their GUI is efficient and interface-rich without being clumsy. No bells and whistles, and it is fine that way.
<br />
<br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwP5U0EHv9RnuLqdQTtrSMcyDlNkWnHk5QncR8EJHvdYk0Kdbn5m6GX9BiAfQOM0bWy5TIvKsA0H3X02WeiRKSyKAxpNOZ58SPmY_R0S2NXH3NjbCccpT2Zc8IrIJ2eFM8TtzNXgIf3Kg/s1600/1.png"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 292px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiwP5U0EHv9RnuLqdQTtrSMcyDlNkWnHk5QncR8EJHvdYk0Kdbn5m6GX9BiAfQOM0bWy5TIvKsA0H3X02WeiRKSyKAxpNOZ58SPmY_R0S2NXH3NjbCccpT2Zc8IrIJ2eFM8TtzNXgIf3Kg/s400/1.png" alt="" id="BLOGGER_PHOTO_ID_5645246137385450370" border="0" /></a><span style="font-size:85%;">The Java-based "Server Manager" does the job efficiently</span>
<br /></div>
<br />Enabling the web-based transfer option was quick and easy to do. What helps is that the software comes with a manual that, without being too detailed, provides lots of screenshots and cookbook-like procedures to configure the server quickly. It took me maybe a minute or so to enable the web server, set up LDAP-based authentication, add a dummy user, and try it out.
<br />
<br />The resulting web service might be bland but it does the job, once again no fireworks. This is very important as it will be deployed to users who might not always be too tech-savvy.
<br />
<br /><div style="text-align: center;"><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUHaCifF_t88-q710BxyhW05MYnDQkc_GaYbx6cUGxZnP6VFgqMxyUkL8ReW8mQvwXGZew7DOrHHNJAykzTBcGJsaahwEid8IbkHVdtspoF8y5ioYDypZYNs-ehp06jLczJjVorI6cSws/s1600/2.PNG"><img style="display: block; margin: 0px auto 10px; text-align: center; cursor: pointer; width: 400px; height: 273px;" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUHaCifF_t88-q710BxyhW05MYnDQkc_GaYbx6cUGxZnP6VFgqMxyUkL8ReW8mQvwXGZew7DOrHHNJAykzTBcGJsaahwEid8IbkHVdtspoF8y5ioYDypZYNs-ehp06jLczJjVorI6cSws/s400/2.PNG" alt="" id="BLOGGER_PHOTO_ID_5645247919440804562" border="0" /></a><span style="font-size:85%;">The web-based service might be simple, that's exactly what I want
<br /></span></div>
<br />
<br />As a system administrator, I would actually <span style="font-style: italic;">want</span> to work with software like this because it's <span style="font-style: italic;">elegant</span>. It does a few things, and it does it well. I like it when software feel natural, and everything works the first time without a glitch. As in every software, I'm sure there are some bugs somewhere, but it sure is a good start.
<br />
<br />Is JSCAPE MFT Server, the iPod of MFT's? I'd say it's not far from it. Chances are that if my project is greenlit, I'll be the first in line to purchase it. Whoever wrote this, good work!
<br />
<br />O.
<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com1tag:blogger.com,1999:blog-1718921294435140810.post-35226931453034029522011-08-12T11:56:00.001-07:002011-08-12T12:07:10.266-07:00Interesting thread on SlashdotSysadmins and developers aren't the same, but they both share a strong technical background.
<br />
<br />I suggest you read this thread. The first comment named "Stay Put" might be hilarious, it makes a lot of sense.
<br />
<br /><a href="http://ask.slashdot.org/story/11/08/12/1433239/Ask-Slashdot-Am-I-Too-Old-To-Learn-New-Programming-Languages">http://ask.slashdot.org/story/11/08/12/1433239/Ask-Slashdot-Am-I-Too-Old-To-Learn-New-Programming-Languages</a>
<br />
<br />The ex-sysadmin I am often asks himself if moving up was a good decision. It turns out it might be after all.
<br />
<br />
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-6008149415079526252011-08-08T18:18:00.000-07:002011-08-08T18:36:55.164-07:00A presentation on IMS and PI might be comingAfter a hiatus in 2010 due to a career change, it's now time to start writing papers and building presentations again. I'll be submitting a paper for Cooper EAS's 2011 <a href="http://www.cooperindustries.com/content/public/en/power_systems/resources/request_forms/smartgrid_registration.html">Smart Grid</a> conference which will consist of my experience with a major <a href="http://www.cooperindustries.com/content/public/en/power_systems/products/automation_and_control/enterprise_software/ied_manager_suiteims.html">Yukon IMS</a> deployment I've been involved with as an IT architect. I'll also explain how we used the SMP gateway to link substations to the <a href="http://www.osisoft.com/">OSISoft PI</a> data historian in order to collect critical data.
<br />
<br />I'm not in academia so my work is in no way scientific. Furthermore, I'm part of a huge team which counts a fair share of people from IT operations, control engineering and electrical engineering, so my view is mostly IT-centric.
<br />
<br />There are many air travel restrictions at the office so I hope I'll be able to make it. The worst case scenario would be driving from Montreal to Minneapolis for which, if it's any consolation, won't require a <a href="http://en.wikipedia.org/wiki/Full_body_scanner">full body scan</a>.
<br />
<br />O.
<br />Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com0tag:blogger.com,1999:blog-1718921294435140810.post-69283994936295204322011-07-04T18:20:00.000-07:002011-07-04T18:47:26.182-07:00Forcing laptop users to use only an Iron Key (and nothing else)I need to transfer files between two networks which need to be physically isolated for a few months until a beefed up and permanent security solution becomes possible.<br /><br />The easiest way to do this on a budget consists of using USB keys to transfer files between two laptops, one which is connected on the intranet, the other on the secure network. If course, the "secure" laptop must be stripped to the bone and have an up-to-date Antivirus so it can trap known viruses that are currently in the wild. That won't prevent any new virus from coming in, but there is an urgent business need to transfer these files so there is not much that can be done in the short term.<br /><br />I'm currently using <a href="https://www.ironkey.com/">IronKeys</a> to ensure the integrity of the data, and also to prevent any data theft if a key is ever stolen. However, one must "encourage" end-users to use these keys, else they might end up using whatever key they lay their hands on to prevent having to enter a password.<br /><br />On Windows XP, there is no way to do USB key filtering based on the key manufacturer. IronKey has a partner named <a href="https://www.ironkey.com/devicelock">DeviceLock</a> that they suggest, but being a commercial product, it comes with a price. There are many other endpoint security tools that can be purchased to offer similar functionality, too. In my case, I was in a deadline and had missed the opportunity acquire software and charge it to the project, so it was preferable to use something free as a stopgap measure.<br /><br />This afternoon, I've been making a few tests with <a href="http://www.simplescripts.de/usb-security.htm">USBSecure</a>. It SEEMS to be free. There is no license, but all code is published so it can be tweaked if necessary. USBSecure is simple to configure: define users that use the computer, and whitelist the device IDs that are allowed on the system. I've been making a few tests for an hour or so, and it seems to work correctly. I might come back and give more details later.<br /><br />Of course, transferring files between two adjacent PCs might look clumsy. But there are a lot of (justified) restrictions on secure control networks. What is sad is that that Stuxnet worked exactly this way, by propagating using USB keys. No matter how much we try to control their usage using endpoint security software, USB keys <span style="font-style: italic;">still</span> remain a vector of infection for secure networks. Better long-term solutions must be done to ensure that any file transferred on a secure network is, indeed, clean. I'll be working on such solutions in 2011-2012.<br /><br />O.Olivier S. Massehttp://www.blogger.com/profile/10917993970005588091noreply@blogger.com1