Amirhossein's Technical Journal

Online Courses & Materials

The Good Parts of AWS

Part 1

• Searching for the optimal option is almost always expensive.

• Instead of searching for the best option, we recommend a technique we call the default heuristic. The premise of this heuristic is that when the cost of acquiring new information is high and the consequence of deviating from a default choice is low, sticking with the default will likely be the optimal choice.

• A default choice is any option that gives you very high confidence that it will work.

• Your default choice doesn’t have to be the theoretical best choice. It doesn’t have to be the most efficient. Or the latest and greatest. Your default choice simply needs to be a reliable option to get you to your ultimate desired outcome. Your default choice should be very unlikely to fail you; you have to be confident that it’s a very safe bet. In fact, that’s the only requirement.

• You would only deviate from your defaults if you realize you absolutely have to.

1. S3

• If you’re storing data—whatever it is—S3 should be the very first thing to consider using.

• It is highly-durable, very easy to use and, for all practical purposes, it has infinite bandwidth and infinite storage space.

• It is also one of the few AWS services that requires absolutely zero capacity management.

• Fundamentally, you can think of S3 as a highly-durable hash table in the cloud. The key can be any string and the value of any blob of data up to 5 TB.

• S3 storage costs $23.55 / TB / month using the default storage class.

• If you’re touching S3 objects at a high frequency (millions of times a day), request pricing becomes an important aspect of S3’s viability for your use case.

• One limitation of S3 is that you cannot append to objects.If you have something that’s changing rapidly (such as alog file), you have to buffer updates on your side for a while, and then periodically flush chunks to S3 as new objects.

• This buffering can reduce your overall data durability, because the buffered data will typically sit in a single place without any replication. A solution for this issue is to buffer data in a durable queue, such as SQS or Kinesis streams, as we’ll see later.

• Not good for hosting a static website.

• Bucket names are globally unique across all AWS customers and across al lAWS regions. A common mitigation is to always add your AWS account ID to the bucket name,which makes conflicts much less likely.

• S3 has an API to check if you own the bucket, and this should always be done before interacting with an existing S3 bucket.

• EC2 allows you to get a complete computer in the cloud in a matter of seconds. The nice thing about EC2 is that the computer you get will be very similar to the computer you use to develop your software.

• If you can run your software on your computer, you can almost certainly run it on EC2 without any changes. This is one of EC2’s main advantages compared to other types of compute platforms (such as Lambda): you don’t have to adapt your application to your host.

• As of the time of writing, EC2 offers 256 different instance types, but they can be narrowed down to a few categories defined by what they’re optimized for: CPU, memory, network, storage, etc., with different instance sizes foreach category.

• One of the most compelling features of EC2 is that you only pay for the number of seconds your instance is running.

• However, EC2 also offers you the option to commit to a long period in exchange for a price reduction.

• There are several cost reduction plans for EC2.

• Amazon will tell you that Auto Scaling allows you to automatically add or remove EC2 instances based on the fluctuating demands of your application.

• As a thought experiment, consider if your EC2 bill were to go down by 30% — would that be a big deal for your business? If not, the effort and complexity of getting Auto Scaling working properly are probably not going to be worth it.

• The other thing to consider is: does your EC2 demand vary enough for Auto Scaling to even matter? If the fluctuations are not significant, or they are too abrupt, or they are not very smooth, Auto Scaling will almost certainly not work well for you.

• Having said all that, you should still almost always use Auto Scaling if you’re using EC2! Even if you only have one instance.

• The other nice thing that comes with Auto Scaling is the ability to simply add or remove instances just by updating the desired capacity setting.

• If EC2 is a complete computer in the cloud, Lambda is a code runner in the cloud.

• With EC2 you get an operating system, a file system, access to the server’s hardware, etc.

• But with Lambda, you just upload some code and Amazon runs it for you. The beauty of Lambda is that it’s the simplest way to run code in the cloud. It abstracts away everything except for a function interface, which you get to fill in with the code you want to run.

• We think Lambda is great — definitely one of the good parts of AWS — as long as you treat it as the simple code runner that it is.

• A problem we often see is that people sometimes mistake Lambda for a general-purpose application host. Unlike EC2, it is very hard to run a sophisticated piece of software on Lambda without making some very drastic changes to your application and accepting some significant new limitations from the platform.

• Lambda is most suitable for small snippets of code that rarely change. We like to think of Lambda functions as part of the infrastructure rather than part of the application. In fact, one of our favorite uses for Lambda is to treat it as a plugin system for other AWS services.

• For example, S3 doesn’t come with an API to resize an image after uploading it to a bucket, but with Lambda, you can add that capability to S3.

• Application load balancers come with an API to respond with a fixed response for a given route, but they can’t respond with an image. Lambda lets you make your load balancer do that.

• CloudFront can’t rewrite a request URL based on request cookies (which is useful for A/B testing), but with Lambda, you can make CloudFront do that with just a little bit of code.

• CloudWatch doesn’t support regex-based alerting on application logs, but you can add that feature with a few lines of Lambda code.

• Kinesis doesn’t come with an API to filter records and write them to DynamoDB, but this is very easy to do with Lambda.

• CloudFormation’s native modeling language has many limitations and, for example, it can’t create and validate a new TLS certificate from the AWS Certificate Manager. Using Lambda, you can extend the CloudFormation language to add (almost) any capability you want.

• Lambda is a great way to extend existing AWS features.

• Treating Lambda as a general-purpose host for your applications is risky. It might look compelling at first — no servers to manage, no operating system to worry about,and no costs when unused—but Lambda’s limitations are insidious hidden risks that typically reveal themselves once your application evolves into something bigger.

• Some limitations will likely improve or go away over time.

• For example, a very annoying issue is the cold start when a function is invoked after a period of inactivity or when Lambda decides to start running your function on new backend workers.

• Another problem is the limit of 250 MB for your code bundle, including all your dependencies.

• And the network bandwidth from Lambda functions seems to be very limited and unpredictable.

• But then there are other limitations that are inherent to the way Lambda works and which are less likely to go away.

• For example, you have to assume that every Lambda invocation is stateless. If you need to access some state, you have to use something like S3 or DynamoDB.

• While this works fine for a demo, it can quickly become prohibitively expensive in the real world. For example, handling a WebSocket connection on Lambda will likely require a read and write to DynamoDB for every exchanged packet, which can quickly result in aspect acularly large DynamoDB bill, even with modest activity.

• Our rule of thumb is simple: If you have a small piece of code that will rarely need to be changed and that needs to run in response to something that happens in your AWS account, then Lambda is a very good default choice.

• Lambda is certainly not a substitute for EC2.

• When using AWS, you almost always want to use some CloudFormation (or a similar tool). It lets you create and update the things you have in AWS without having to click around on the console or write fragile scripts.

• It takes a while to get the hang of it, but the time savings pay off the initial investment almost immediately. Even for development, the ability to tear down everything cleanly and recreate your AWS setup in one click is extremely valuable.

• With CloudFormation, you define your AWS resources as a YAML script (or JSON, but we find YAML to be much easier to read and modify).

• Then you point CloudFormation to your AWS account, and it creates all the resources you defined.

• Our rule of thumb is to let CloudFormation deal with all the AWS things that are either static or change very rarely; things such as VPC configurations, security groups, load balancers, deployment pipelines, and IAM roles.

• Other things, such as DynamoDB tables, Kinesis streams, Auto Scaling settings, and sometimes S3 buckets,are better managed elsewhere.

• SQS is a highly-durable queue in the cloud. You put messages on one end, and a consumer takes them out from the other side. The messages are consumed in almost first-in-first-out order, but the ordering is not strict.

• You can think of a Kinesis stream as a highly-durable linked list in the cloud. The use cases for Kinesis are often similar to those of SQS — you would typically use either Kinesis or SQS when you want to enqueue records for asynchronous processing. The main difference between the two services is that SQS can only have one consumer, while Kinesis can have many.

AWS Certified Cloud Practitioner 2020

Objectives

• Cloud Concepts

• Security

• Technology

• Billing and Pricing

• Compute —> EC2, Lambda

• Storage —> Simple Storage Service (S3), Glacier

• Databases —> Relational Database Service (RDS), DynamoDB (Non Relational Databases)

• Migration & Transfer

• Network & Content Delivery —> VPC, Route53

• Security, Identity & Compliance

• AWS Cost Management

1. Data Sovereignty Laws

2. Latency to end users

3. AWS Services

Create A Billing Alarm- LAB —> CloudWatch

Let's Start To Clound! Identity Access Management (IAM)- LAB

• IAM stands for Identity Access Managements. It is Global, you do not specify a region when dealing with IAM. When you create a user or group, this is created GLOBALLY.

• You can access the AWS platform in 3 ways: 1- Via the Console. 2- Programmatically (Using the Command Line). 3- Using the Software Developers Kit (SDK).

• Your root account is the email address you used to set up your AWS account. The root account always has full administrator access. You should not give these account credentials away to anyone. Instead create a user for each individual within your organization. You should always secure this root account using multi-factor authentication.

• A group is simply a place to store your users. Your users will inherit all permissions that the group has. Examples of group might be developers, system administrators, human resources, finance etc.

• To set the permissions in a group you need to apply a policy to that group. Policies consist of JSON. These are referred to as key-value pairs.

IAM Best Practices

• Root Account: Only use the root account to create your AWS account. Don not use it to log in.

• Users: One user should equal one real human being. Don't create phantom users.

• User/Groups/Policies: Always place users in groups, and then apply policies to the groups. This makes management easier.

• Passwor Policies: Have a strong password rotation policy.

• MFA: Always enable MFA wherever possible.

• Roles: Use roles to access various other AWS services.

• Access Keys: Use access keys for programmatic access to AWS.

• IAM Credential Report: Use IAM credential reports to audit the permissions of your users/accounts.

IAM Credential Reports

S3 101

• S3 provides developers and IT teams with secure, durable, highly-scalable object storage. Amazon S3 is easy to use, with a simple web services interface to store and retrieve any amount of data from anywhere on the web.

• S3 is a safe place to store your files.

• It is Object-based storage —> i.e., allows you to upload files.

• Objects consist of the following: 1- Key (This is simply the name of the object) 2- Value (This is simply the data and is made up of a sequences of bytes) 3- Version ID (Important for versioning) 4- Metadata (Data about data you are storing)
5- Subresources —> Access Control List, Torrent

• The data is spread across multiple devices and facilities.

• Files can be from 0 Bytes to 5TB.

• There is unlimited storage.

• Files are stored in Buckets.

• S3 is a universal namespace. That is, names must be unique globally.

• When you upload a file to S3, you will receive a HTTP 200 code if the upload was successful.

• Read after Write consistency for PUTS of new Objects. —> If you write a new file and read it immediately afterwards, you will be able to view that data.

• Eventual Consistency for overwrite PUTS and DELETES (can take some time to propagate) —> If you update an existing file or delete a file and read it immediately, you may get the older version, or you may not. Basically changes to objects can take a bit of time to propagate.

• Tiered Storage Available

• Lifecycle Management

• Versioning

• Encryption

• Secure your data using Access Control Lists (work on individual files) and Bucket Policies (across entire bucket)

• Storage

• Requests

• Storage Management Pricing

• Data Transfer Pricing

• Transfer Acceleration —> Amazon S3 Transfer Acceleration enables fast, easy, and secure transfer of files over long distances between your end users and an S3 bucket. Transfer Acceleration takes advantage of Amazon CloudFront's globally distributed edge locations. As the data arrives at an edge location, data is routed to Amazon S3 over an otimized network path.

• Cross Region Replication Pricing

Let's Create an S3 Bucket!- LAB

• Bucket names share a common name space. You cannot have the same bucket name as someone else.

• When you view your buckets you view them globally but you can have buckets in individual regions.

• Bucket Policies- Applies across the whole bucket.

• Object Policies - Applies to individual files.

• IAM Policies to Users & Groups - Applies to Users & Groups.

Let's Create an S3 Website! - LAB

• You can use bucket policies to make entire S3 buckets public.

• You can use S3 to host STATIC websites (such as .html). Websites that require database connections such as Wordpress etc cannot hosted on S3.

• S3 scales automatically to meet your demand. Many enterprises will put static websites on S3 when they think there is going to be a large number of requests (such as for a movie preview for example)

S3 versioning

• Stores all versions of an object —> Including all writes and even if you delete an object.

• Great backup tool

• Versioning cannot be disabled —> Once enabled, versioning cannot be disabled (only suspended)

• Integrates with lifecycle rules

• Versioning MFA capability —> Uses multi-factor authentication; can be used to provide an additional layer of security.

Packt- Hands-On Machine Learning Using Amazon SageMaker

Your First Machine Learning Model on SageMaker

• The Course Overview

• AWS Setup

• What Problem You Will Solve

• Train The Model on SageMaker

• Deploy The Model as a REST service on SageMaker

• Sign in to the AWS Management Console as an IAM user and open the IAM console at https://console.aws.amazon.com/iam/

• Select Roles from the list in the left-hand side, and click on Create role

• Then, select SageMaker.

• Click Next: Review on the following page.

• Type a name for the SageMaker role, i.e. PacktSageMaker, and click on Create role.

• Click on the created role and, then, click on Attach policy and search for AmazonEC2ContainerRegistryFullAccess. Attach the corresponding policy.

• Do the same to attach the AmazonS3FullAccess and IAMReadOnlyAccess policies, and end up with the following.

• Now, go to Users page by clicking on Users on the left-hand side.

• Click on Add user and type packt-sagemaker as username.

• Copy the ARN of that user.

• Then, go back the page of the Role you created and click on the Trust relationships tab.

• Click on Edit trust relationship and add the following

• You're almost there! Make sure that you have added the IAM user in your ~/.aws/credentials file. For example:

• And, finally, add the following in the ~/.aws/config file:

• Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables

Word2Vec

Papers

Efficient Estimation of Word Representations in Vector Space- T Mikolov 2013

• For all the following models, the training complexity is proportional to:

• The probabilistic feedforward neural network language model has been proposed in [Y. Bengio, A neural probabilistic language mode].

• It consistsof input, projection, hidden and output layers.

• At the input layer, previous words are encoded using coding, where is the size of the vocabulary.

• The input layer is then projected to a projection layer that has dimensionality , using a shared projection matrix.

• The NNLM architecture becomes complex for computation between the projection and the hidden layer, as values in the projection layer are dense. For a common choice of , the size of the projection layer () might be to while the hidden layer size is typically to units.

• Moreover, the hidden layer is used to compute a probability distribution over all the words in the vocabulary, resulting in an output layer with dimensionality .

• The computational complexity per each training example is , where the dominating term is .

• However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [referenced] or avoiding normalized models completely by using models that are not normalized during training [referenced]. With binary tree representations of the vocabulary, the number of output units that need to be evaluated can go down to around . Thus, most of the complexity is caused by the term .

• In our models, we use hierarchical softmax where the vocabulary is represented as a Huffman binary tree. Huffman trees assign short binary codes to frequent words, and this further reduces the number of output units that need to be evaluated.

• Recurrent neural network based language model has been proposed to overcome certain limitations of the feedforward NNLM, such as the need to specify the context length (the order of the model ), and because theoretically, RNNs can efficiently represent more complex patterns than the shallow neural networks [referenced].

• The complexity per training example of the RNN model is where the word representations have the same dimensionality as the hidden layer .

• Again the hierarchical softmax is used.

Distributed Representations of Words and Phrases and their Compositionality- T Mikolov 2013

• By subsampling of the frequent words, we obtain significant speed up and also learn more regular word representations.

• We also describe a simple alternative to the hierarchical softmax called negative sampling.

• The Skip-gram model architecture. The training objective is to learn word vector representations that are good at predicting the nearby words.

• Unlike most of the previously used neural network architectures for learning word vectors, training of the Skip-gram model (figure above) does not involve dense matrix multiplications. This makes the training extremely efficient: an optimized single-machine implementation can train on more than 100 billion words in one day.

• In this paper, we present several extensions of the original Skip-gram model. We show that subsampling of frequent words during training results in a significant speedup (around 2x - 10x), and improves the accuracy of the representations of less frequent words.

• In addition, we present a simplified variant of Noise Contrastive Estimation (NCE) [Michael U Gutmann- Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.] for training the Skip-gram model that results in faster training and better vector representations for frequent words, compared to more complex hierarchical softmax that was used in the prior work.

• The training objective of the Skip-gram model is to find word representations that are useful for predicting the surrounding words in a sentence or a document.

• More formally, given a sequence of training words , the objective of the Skip-gram model is to maximize the average log probability:

• Larger results in more training examples and thus can lead to higher accuracy, at the expense of the training time.

• The basic Skip-gram formulation defines using the softmax function:

• Where and are the “input” and “output” vector representations of , and is the number of words in the vocabulary.

• This formulation is impractical because the cost of computing is proportional to , which is often large ( terms).

• A computationally efficient approximation of the full softmax is the hierarchical softmax.

• The main advantage is that instead of evaluating output nodes in the neural network to obtain the probability distribution, it is needed to evaluate only about nodes.

• The hierarchical softmax uses a binary tree representation of the output layer with the words as its leaves and, for each node, explicitly represents the relative probabilities of its child nodes. These define a random walk that assigns probabilities to words.

• More precisely, each word can be reached by an appropriate path from the root of the tree. Let be the node on the path from the root to and let be the length of this path, so and . In addition, for any inner node , let be an arbitrary fixed child of and let be if is true and otherwise. Then the hierarchical softmax defines as follows:

• In our work we use a binary Huffman tree, as it assigns short codes to the frequent words which results in fast training.

• An alternative to the hierarchical softmax is Noise Contrastive Estimation (NCE).

• NCE posits that a good model should be able to differentiate data from noise by means of logistic regression.

• We define Negative sampling (NEG) by the objective:

• Our experiments indicate that values of in the range are useful for small training datasets, while for large datasets the can be as small as .

• The main difference between Negative sampling and NCE is that NCE needs both samples and the numerical probabilities of the noise distribution, while Negative sampling uses only samples.

• In very large corpora, the most frequent words can easily occur hundreds of millions of times (e.g.,“in”, “the”, and “a”). Such words usually provide less information value than the rare words.

• The vector representations of frequent words do not change significantly after training on several million examples.

• To counter the imbalance between the rare and frequent words, we used a simple subsampling approach: each word in the training set is discarded with probability computed by the formula , where is the frequency of word and is a chosen threshold, typically around .

• We chose this subsampling formula because it aggressively subsamples words whose frequency is greater than while preserving the ranking of the frequencies.

• Although this subsampling formula was chosen heuristically, we found it to work well in practice. It accelerates learning and even significantly improves the accuracy of the learned vectors of the rare words.

• The subsampling of the frequent words improves the training speed several times and makes the word representations significantly more accurate.

Video Lecture

• Distributional similarity based representations: you can get a lot of value by representing a word utilizing its neighbors (it's context). "You shall know a word by the company it keeps. (J.R. Firth 1957)"

• The idea of the skip-gram model is for each estimation step, you are taking one word as the center word and then you are going to try and predict words in their context out to some window size. And so the model is going to define a probability distribution that is the probability of a word appearing in the context given the center word. And we are going to choose vector representations of words so we can try and maximize that probability distribution. We just have one probability distribution of a context word, which we'll refer to as the output, occurring in the context close to the center word. All the parameters of this model are vector representations of each word. we actually have two representations for each word, one when the word is used as the center word and one when the word is used as a context word.

• All the parameters of this model are vector representations of words. There are actually two vector representations for each word; one as a center word and the other as a context word. .

• first term: second term: (chain rule) final form: first term is what we observed, the actual output context word appeared and the second term has the form of a expectation with the probability of every possible word appearing in the context and based on that probability we get taking that much of that , so this is in some sense, the expectation vector, the average over all the possible context vectors weighted by their likelihood of occurrence.

• We are also going to calculate the partial derivatives with respect to the context vectors.

Implementation

Problem Statement

• Let be the set of users.

• Let be the of items.

• Each item has features, composed of numerical features and categorical features ().

• For each user, we have the sequence of his/her historical viewed items, i.e., , where is the sequence of historical viewed items of user and is the number of his/her viewed items.

• Compute a dense representation of each item such that each item's dense vector representation contains enough information content that is good at predicting the nearby items in users' sessions.

CarNext Products Dense Representation Learning Proposal

• For each user, extract the sequence of product_ids

• Write a dataloader that have the following functionalities given two arguments (context_window_size, num_negative_samples): 1- extract a context window including 'context_window_size' product_ids from all the sequences. 2- for any context_window, extract 'num_negative_samples' random negative samples

• Write a model to embed product_ids into a dense space (e.g., dimensionality )

• Implement and serve the model in SageMaker.

Graph Convolutional Matrix Completion

Paper

Introduction

• We consider matrix completion for recommender systems from the point of view of link prediction on graphs. Interaction data such as movie ratings can be represented by a bipartite user-item graph with labeled edges denoting observed ratings.

• Content information can naturally be included in this framework in the form of node features. Predicting ratings then reduces to predicting labeled links in the bipartite user-item graph.

• We propose graph convolutional matrix completion (GC-MC): a graph-based auto-encoder framework for matrix completion, which builds on recent progress in deep learning on graphs.

• The auto-encoder produces latent features of user and item nodes through a form of message passing on the bipartite interaction graph. These latent user and item representations are used to reconstruct the rating links through a bi-linear decoder.

• The benefit of formulating matrix completion as a link prediction task on a bipartite graph becomes especially apparent when recommender graphs are accompanied with structured external information such as social networks. Combining such external information with interaction data can alleviate performance bottlenecks related to the problem.

Matrix completion as link prediction in bipartite graphs

• Consider a rating matrix of shape , where is the number of users and is the number of items. Entries in this matrix encode either an observed rating (user rated item ) from a set of discrete possible rating values or the fact that the rating is unobserved (encoded by the value ). The task of matrix completion or recommendation can be seen as predicting the value of unobserved entries in .

• In an equivalent picture, matrix completion or recommendation can be cast as a link prediction problem on a bipartite user-item interaction graph.

• More precisely, the interaction data can be represented by an undirected graph with entities consisting of a collection of user nodes with , and item nodes with , such that . The edges carry labels that represent ordinal rating levels, such as .

• Graph auto-encoders are comprised of: 1- a graph encoder model , which take as input an feature matrix and a graph adjacency matrix , and produce an node embedding matrix . 2- a pairwise decoder model , which takes pairs of node embedding and predicts respective entries in the adjacency matrix. Note that denotes the number of nodes, the number of input features, and the embedding size.

• For bipartite recommender graphs , we can reformulate the encoder as , where is the adjacency matrix associated with rating type , such that contains ’s for those elements for which the original rating matrix contains observed ratings with value . and are now matrices of user and item embeddings with shape and , respectively. A single user (item) embedding takes the form of a real-valued vector () for user (item ).

• Analogously, we can reformulate the decoder as , i.e. as a function acting on the user and item embeddings and returning a (reconstructed) rating matrix of shape . We can train this graph auto-encoder by minimizing the reconstruction error between the predicted ratings in and the observed ground-truth ratings in . Examples of metrics for the reconstruction error are the root mean square error, or the cross-entropy when treating the rating levels as different classes.

• In what follows, we propose a particular choice of encoder model that makes efficient use of weight sharing across locations in the graph and that assigns separate processing channels for each edge type (or rating type) .

• In our case, we can assign a specific transformation for each rating level, resulting in edge-type specific messages from items to users of the following form: (1)

• Here, is a normalization constant, which we choose to either be (left normalization) or (symmetric normalization) with denoting the set of neighbors of node . is an edge-type specific parameter matrix and is the (initial) feature vector of node . Messages from users to items are processed in an analogous way. After the message passing step, we accumulate incoming messages at every node by summing over all neighbors under a specific edge-type , and by subsequently accumulating them into a single vector representation: (2) where denotes an accumulation operation, such as , i.e. a concatenation of vectors (or matrices along their first dimension), or , i.e. summation of all messages. denotes an element-wise activation function such as the .

• To arrive at the final embedding of user node , we transform the intermediate output as follows: (3)

• The item embedding is calculated analogously with the same parameter matrix . In the presence of user- and item-specific side information we use separate parameter matrices for user and item embeddings. We will refer to (2) as a graph convolution layer and to (3) as a dense layer. Note that deeper models can be built by stacking several layers (in arbitrary combinations) with appropriate activation functions. In initial experiments, we found that stacking multiple convolutional layers did not improve performance, and a simple combination of a convolutional layer followed by a dense layer worked best.

• It is worth mentioning that the model demonstrated here is only one particular possible, yet relatively simple choice of an encoder, and other variations are potentially worth exploring. Instead of a simple linear message transformation, one could explore variations where is a neural network in itself. Instead of choosing a specific normalization constant for individual messages, such as done here, one could employ some form of attention mechanism, where the individual contribution of each message is learned and determined by the model.

• For reconstructing links in the bipartite interaction graph we consider a bi-linear decoder and treat each rating level as a separate class. Indicating the reconstructed rating between user and item with , the decoder produces a probability distribution over possible rating levels through a bi-linear operation followed by the application of a softmax function: . (4)
with a trainable parameter matrix of shape , and the dimensionality of the hidden user (item) representations (). The predicted rating is computed as:

• In practice, we can use efficient sparse matrix multiplications, with complexity linear in the number of edges, i.e. , to implement the graph auto-encoder model. The graph convolutional encoder (Eq. 3), for example in the case of left normalization, can be vectorized as follows:

• The summation in (8) can be replaced with concatenation, similar to (2). In this case denotes the diagonal node degree matrix with nonzero elements . Vectorization for an encoder with a symmetric normalization, as well as vectorization of the bilinear decoder, follows in an analogous manner. Note that it is only necessary to evaluate observed elements in , given by the mask in Eq. 6.

Implementation

Introduction

/src

• w2v_item_embeddings: A torch tensor of the shape () of word2vec item embeddings, which is used as the side information of item nodes.

• edge_index: edge indices of adjacency matrices of the three classes of bipartite graphs.

• edge_type: class type of each edge.

• x_identity: an identity matrix of the size of the total nodes of the graph (#users + #items), which is used as the initial feature matrix of the nodes of the graph.

• users_norm_r_list: A list containing three tensors that are used for left normalization of the messages that are passed from item nodes to user nodes.

• items_norm_r_list: A list containing three tensors that are used for left normalization of the messages that are passed from user nodes to item nodes.

• users_one_hot_degree: A tensor containing one-hot user degrees of each bipartite graph, which is used as the side information of user nodes.

• train_val_test_indices: A dictionary with train, validation and test masks.

train

inference

Dockerfile

executor.sh

build.sh

push.sh

local_train.sh

local_deploy.sh

/local_test

Train and Serve the Model on AWS SageMaker

Next Steps

Refactoring the code

Handling the zero class problem

• the class probabilities of the the dataset are p(*)=0.7742, p(**)=0.1645, p(***)=0.0612.

• the weights=[1.26, 6.07, 16.33] of each class is then computed as: w(*)=(1/p(*))*100=1.26, w(**)=(1/p(**))*100=6.07, w(***)=(1/p(***))*100=16.33

• the weights are passe to the CrossEntropLoss function. here is the results:

Productionizing the code

Creating a blueprint for ML projects