Graph Embedding for Recommendation against Attribute Inference Attacks

In recent years, recommender systems play a pivotal role in helping users identify the most suitable items that satisfy personal preferences. As user-item interactions can be naturally modelled as graph-structured data, variants of graph convolutional networks (GCNs) have become a well-established building block in the latest recommenders. Due to the wide utilization of sensitive user profile data, existing recommendation paradigms are likely to expose users to the threat of privacy breach, and GCN-based recommenders are no exception. Apart from the leakage of raw user data, the fragility of current recommenders under inference attacks offers malicious attackers a backdoor to estimate users' private attributes via their behavioral footprints and the recommendation results. However, little attention has been paid to developing recommender systems that can defend such attribute inference attacks, and existing works achieve attack resistance by either sacrificing considerable recommendation accuracy or only covering specific attack models or protected information. In our paper, we propose GERAI, a novel differentially private graph convolutional network to address such limitations. Specifically, in GERAI, we bind the information perturbation mechanism in differential privacy with the recommendation capability of graph convolutional networks. Furthermore, based on local differential privacy and functional mechanism, we innovatively devise a dual-stage encryption paradigm to simultaneously enforce privacy guarantee on users' sensitive features and the model optimization process. Extensive experiments show the superiority of GERAI in terms of its resistance to attribute inference attacks and recommendation effectiveness.


ABSTRACT
In recent years, recommender systems play a pivotal role in helping users identify the most suitable items that satisfy personal preferences. As user-item interactions can be naturally modelled as graph-structured data, variants of graph convolutional networks (GCNs) have become a well-established building block in the latest recommenders. Due to the wide utilization of sensitive user profile data, existing recommendation paradigms are likely to expose users to the threat of privacy breach, and GCN-based recommenders are no exception. Apart from the leakage of raw user data, the fragility of current recommenders under inference attacks offers malicious attackers a backdoor to estimate users' private attributes via their behavioral footprints and the recommendation results. However, little attention has been paid to developing recommender systems that can defend such attribute inference attacks, and existing works achieve attack resistance by either sacrificing considerable recommendation accuracy or only covering specific attack models or protected information. In our paper, we propose GERAI, a novel differentially private graph convolutional network to address such limitations. Specifically, in GERAI, we bind the information perturbation mechanism in differential privacy with the recommendation capability of graph convolutional networks. Furthermore, based on local differential privacy and functional mechanism, we innovatively devise a dual-stage encryption paradigm to simultaneously enforce privacy guarantee on users' sensitive features and the model optimization process. Extensive experiments show the superiority of GERAI in terms of its resistance to attribute inference attacks and recommendation effectiveness.

CCS CONCEPTS
• Information systems → Collaborative filtering.

INTRODUCTION
With the explosive growth of e-commerce, consumers are shopping with online platforms more frequently [10,20,54]. As an effective solution to information overload, recommender systems automatically discover the most relevant items or services for each user and thus improve both the user experience and business revenue. For this reason, recommender systems have become an indispensable part in our contemporary lives.
Latent factor models like matrix factorization [35] are typical collaborative filtering-based recommendations, which infer user-item interactions via learned latent user/item representations. Because user-item interactions can be conveniently formulated as graphstructured data, graph embedding-based recommenders [42,52,59] are highly effective in uncovering users' subtle preferences toward items. As deep neural networks demonstrate superior capability of representation learning in various machine learning tasks, deep recommendation models, especially those derived from graph convolutional networks (GCNs) [49,51,55,56] have recently become one of the most prominent techniques in this field.
To enhance the recommendation performance, especially for fresh (i.e., cold-start) customers, it is a common practice to incorporate side information (a.k.a. features or contexts) [2,47,58] about users. During user registration, some service providers even start persuading users to complete questionnaires about personal demographics to facilitate user profiling. However, the utilization of user data containing personal information often sparks serious privacy concerns. A 2018 survey [26] showed that more than 80% US Internet users were concerned about how their personal data is being used on Facebook; and among Facebook users sharing less content on social media, 47% reported that privacy issue was the main concern. Consequently, with the growing public awareness on privacy, a dilemma is presented to e-commerce platforms: either they proceed with such sensitive data acquisition process despite the high risk on privacy breach, or they allow users not to disclose their sensitive attributes but provide compromised recommendation performance as a result. In that sense, a sound privacy guarantee on the user side is highly desirable, which avoids uploading the unencrypted raw user features to a recommender system. Furthermore, according to the example that Apple is now telling users their personal data is protected before being shared for analytics, it also helps increase users' willingness to share their sensitive data.
Meanwhile, a more critical privacy issue comes from the fact that users' sensitive attributes can still be disclosed purely based on how they behave. Regardless of the availability of features, recommenders learn explicit or latent profiles that reflect users' preferences based on her/his behavioral footprints (e.g., previous ratings and reviews), and produce personalized recommendations with the constructed profiles [40]. However, many early studies have shown that even a user's personal information can be accurately inferred via her/his interaction history [5,29,50]. Such personal information includes age, gender, political orientation, health, financial status etc. and are highly confidential. Furthermore, the inferred attributes can be utilized to link users across multiple sites and break anonymity [16,44]. For example, [36] successfully deanonymizes Netflix users using the public IMDb user profiles.
Due to the open-access nature of many platforms (e.g., Yelp and Amazon), users' behavioral trajectories can be easily captured by a malicious third-party, leading to catastrophic leakage of inferred user attributes. This is known as the attribute inference attack [17], where the malicious attackers can be cyber criminals, data brokers, advertisers, etc. By proving that even a person's racial information and sexual orientation can be precisely predicted from merely the "like" behaviors on Facebook, Kosinski et al. [29] demonstrated that users' preference signals are highly vulnerable to attribute inference attacks. This is especially alarming for many GCN-based recommenders, since user representations are usually formed by aggregating information from her/his interacted items. Moreover, the personalized recommendation results can also be utilized by attackers since they are strong reflections on users' preferences and are increasingly accessible via services like friend activity tracing (e.g., Spotify) and group recommendation [54]. Hence, this motivates us to design a secure recommender system that stays robust against attribute inference attacks.
In GCN-based recommenders, graphs are constructed by linking user and item nodes via their interactions. However, though existing GCNs are advantageous in binding a node's own features and its high-order connectivity with other nodes into an expressive representation, they exhibit very little consideration on user privacy. In fact, the field of privacy-preserving recommender systems that are resistant to attribute inference attacks is far from its maturity. [6,14,37,39] have applied cryptography algorithms to the recommendation models, but the computational cost of encryption is too high to support real-world deployment. Recently, the notion of differential privacy (DP) has become a well-established approach for protecting the confidentiality of personal data. Essentially, DP works by adding noise to each data instance (i.e., perturbation), thus masking the original information in the data. In the context of both recommendation and graph embedding, there has also been attempts to adopt DP to perturb the output of matrix factorization algorithms [4,33,53]. Unfortunately, these approaches are designed to only prevent membership attacks which infer users' real ratings in the dataset, and are unable to provide a higher level of protection on users' sensitive information against inference attacks. A recent work [3] systematically investigates the problem of developing and evaluating recommender systems under the attribute inference attack setting. Their proposed model RAP [3] utilizes an adversarial learning paradigm where a personalized recommendation model and an attribute inference attack model are trained against each other, hence the attackers are more likely to fail when inferring user attributes from interaction records. However, it suffers from two major limitations. Firstly, as the design of RAP requires a pre-specified and fixed attribute inference model, its resistance to any arbitrary attacker is unguaranteed given the unpredictability of the inference model that an attacker may choose. Secondly, though RAP assumes the existence of users' sensitive attributes, it only treats them as ground-truth labels for training the inference model, and does not incorporate such important side information for recommendation. This design not only fails to ease users' privacy concerns on submitting their original attributes, but also greatly hinders the model's ability to securely utilize user features to achieve more accurate recommendation results.
To this end, we address a largely overlooked defect of existing GCN-based recommenders, i.e., protecting users' private attributes from attribute inference attacks. Meanwhile, unlike existing inference-resistant recommenders, we would like the model to take advantage of user information for accurate recommendation without exerting privacy breach. In this paper, we subsume the GCN-based recommender under the differential privacy (DP) constraint, and propose a novel privacy-preserving recommender GERAI, namely Graph Embedding for Recommendation against Attribute Inference Attacks. In GERAI, we build its recommendation module upon the state-of-the-art inductive GCNs [11,21,28] to jointly exploit the user-item interactions and the rich side information of users. To achieve optimal privacy strength, we propose a novel dual-stage perturbation paradigm with DP. Firstly, at the input stage, GERAI performs perturbation on the raw user features. On one hand, this offers users a privacy guarantee while sharing their sensitive data. On the other hand, the perturbed user features will make the generated recommendations less dependent on a user's true attributes, making it harder to infer those attributes via recommendation results. Specifically, we introduce local differential privacy (LDP) for feature perturbation, where each individual's original feature vector is transformed into a noisy version before being processed by the recommendation module. We further demonstrate that the perturbed input data satisfies the LDP constraint while retaining adequate utility for the recommender to learn the subtle user preferences. Secondly, we enforce DP on the optimization stage of GERAI so that the recommendation results are less likely to reveal a user's attributes and preferences [3,4,33] in the inference attack. To achieve this, we innovatively resort to the functional mechanism [57] that allows to enforce DP by perturbing the loss function in the learning process. Different from methods that applies perturbation on recommendation results [4], by perturbing the loss function, GERAI defends the inference attack without setting obstacles for learning meaningful associations between user profiles and recommended items.
Overall, we summarize our contributions in the following: • We address the increasing privacy concerns in the recommendation context, and propose a novel solution GERAI, namely differentially private graph convolutional network to protect users' sensitive data against attribute inference attacks and provide high-quality recommendations at the same time. • Our proposed GERAI innovatively incorporates differential privacy with a dual-stage perturbation strategy for both the input features and the optimization process. As such, GERAI assures user privacy and offers better recommendation effectiveness than existing privacy-preserving recommenders. • We conduct extensive experiments to evaluate the performance of GERAI on real-world data. Comparisons with stateof-the-art baselines show that GERAI provides a better privacy guarantee with less compromise on the recommendation accuracy.

PRELIMINARIES
In this section, we first revisit the definitions of differential privacy and then formally define our problem. Note that in the description below, all vectors and matrices are respectively denoted wiht bold lowercase and bold uppercase letters, and all sets are written in calligraphic uppercase letters. Differential Privacy. Differential privacy (DP) is a strong mathematical guarantee of privacy in the context of machine learning tasks. DP was first introduced by [13] and it aims to preclude adversarial inference on any raw input data from a model's output. Given a privacy coefficient > 0, the −differential privacy ( −DP) is defined as follows: Definition 2.1. ( −Differential Privacy) For a randomized function (e.g., a perturbation algorithm or machine learning model) (·) that takes a dataset as its input, it satisfies −DP if: where [·] represents probability, D and D ′ are any two datasets differing on only one data instance, and denotes all subsets of possible output values that (·) produces. If is continuous, then the probability term can be replaced by a probability density function. Eq.(1) implies that the probability of generating the model output with D is at most ( ) times smaller than with D ′ . That is, (·) should not overly depend on any individual data instance, providing each instance roughly the same privacy. As a common practice for privacy protection, each individual user's personal data can be perturbed by adding controlled noise before it is fed into (·). In this case, the data owned by every user is regarded as a singleton dataset, and we require the function (·) to provide differential privacy when such a singleton database is given as the input. Specifically, this is termed as −local differential privacy ( −LDP): Definition 2.1. ( −Local Differential Privacy) A randomized function (·) satisfies −LDP if and only if for any two users' data and ′ , we have: where * denotes the output of (·). The lower provides stronger privacy but may result in lower accuracy of a trained machine learning model as each user's data is heavily perturbed. Hence, is also called the privacy budget that controls the trade-off between privacy and utility in DP. With the security guarantee from DP, an external attacker model cannot infer which user's data is used to produce the output * (e.g., the recommendation results) with high confidence. Privacy-Preserving Recommender System. Let G = (U ∪ V, E) denote a weighted bipartite graph. U = { 1 , 2 , ..., |U | } and V = { 1 , 2 , ..., |V | } are the sets of users and items. A weighted edge ( , , ) ∈ E means that user has rated item , with weight as 1. We use N ( ) to denote the set of items rated by and N ( ) to denote all users who have rated item . Following [3], for each user we construct a dense input vector x ∈ R 0 with each element representing either a sensitive attribute ∈ S or a pre-defined statistical feature ∈ S ′ of . All categorical features are represented by one-hot encodings in x , while all numerical features are further normalized into [−1, 1]. We define the target of a privacy-preserving recommender system below.
Problem 1. Given the weighted graph G and user feature vectors {x | ∈ U}, we aim to learn a privacy-preserving recommender system that can recommend products of interest to each user, while any malicious attacker model cannot accurately infer users' sensitive attributes (i.e., gender, occupation and age in our case) from the users' interaction data including both the users' historical ratings and current recommendation results. It is worth noting that our goal is to protect users against a malicious attacker, but not against the recommender system that is trusted.

GCN-BASED RECOMMENDATION MODULE
As we aim to address the privacy concerns in GCN-based recommendation models, in this work we build our base recommender upon GCNs [21,28]. A recommender, at its core, learns vector representations (a.k.a. embeddings) of both users and items based on their historical interactions, then a user's interest on each item can be easily inferred by measuring the user-item similarity in the latent vector space. When performing recommendation on the graph-structured data, owing to the ability to preserve a graphs topological structure, GCNs can produce highly expressive user and item embeddings for recommendation. Given a weighted graph G = (U ∪ V, E), users and items are two types of nodes connected by observed links. Then, for each node, GCN computes its embedding by iteratively aggregating information from its local neighbors, where all node embeddings are optimized for predicting the affinity of each user-item pair for personalized ranking.
We first introduce our recommendation module from the user side. For each user , the information I ( ) passed into comes from the user's first-order neighbors, i.e., items rated by : where (·) is a multi-layer perceptron, m /m are the messages from user/item nodes, and z , z ∈ R respectively denote the learnable latent embeddings of user and item . Note that z , z can be initialized as follows: where x ∈ R 0 is user 's raw feature vector and E U ∈ R × 0 is the user embedding matrix. x ∈ R 1 and E V ∈ R × 1 are respectively the item feature vector and embedding matrix. To where (·) is the aggregation function and (·) denotes the rectified linear unit for nonlinearity, and W and b are learnable weight matrix and bias vector. Motivated by the effectiveness of attention mechanism [45] in graph representation learning, we quantify the varied contributions of each element in I ( ) to embedding z * by assigning each neighbour node a different weight. Formally, we define (I ( )) as: where denotes the attention weight implying the importance of message m ∈ I ( ) to user node during aggregation. Specifically, to compute , we first calculate an attention score via the following attention network: where ⊕ represents the concatenation of two vectors. Afterwards, each final attention weight is computed by normalizing all the attentive scores using softmax: Likewise, on the item side, we repeat the message passing scheme by aggregating the information from an item's interacted users in N ( ) to learn the item embedding z * : where Note that the same network structure and trainable parameters are shared in the computation of both user and item embeddings.
To train our model for top-recommendation, we leverage the pairwise Bayesian personalized ranking (BPR) loss [41] to learn model parameters. To facilitate personalized ranking, we firstly generate a ranking score for an arbitrary user-item tuple ( , ): where h ∈ R is the projection weight. Intuitively, BPR optimizes ranking performance by comparing two ranking scores , ′ for user on items and ′ . In each training case ( , , ′ ), is the positive item sampled from E, while ′ is the negative item having ′ ∉ E. Then, BPR encourages that should have a higher ranking score than ′ by enforcing: where D is the training set, (·) is the sigmoid function, Θ denotes parameters in the GCN-based recommendation module, and is the L2-regularization coefficient.
Uniformly sample from [− , ℓ ( ) ∪ ( ), ]; end return differential privacy ( ▷ −LDP) by directly adding noise to users' raw feature vectors x used for learning user embeddings, which can avoid exposing users' sensitive data to an unsecured cyber environment during upload, while providing the GCN-based recommender with side information for learning expressive user representations. Then, to prevent GERAI from generating recommendation results that can reveal users' sensitive attributes, we further enforce −DP in the optimization stage by perturbing its loss function L. However, this is a non-trivial task as it requires to calculate the privacy sensitivity of L, which involves analyzing the complex relationship between the input data and learnable parameters. Hence, we propose a novel solution by deriving a polynomial approximation L of the original BPR loss L, so as to support sensitivity calculation and perform perturbation on L to facilitate differentially private training of GERAI. Notably, to distinguish the DP constraints in two stages, we denote ▷ as local privacy budget and as global privacy budget, respectively.

User Feature Perturbation at Input Stage
At the input level, the feature vector x of each user is perturbed before being fed into the recommender module. This helps address users' privacy concerns on sharing their personal attributes and keep them confidential during the upload process. Furthermore, as we will show in Section 5.7, perturbing user features contributes to defending attribute inference attacks as the recommendation results are no longer based on the actual attributes. Then, instead of the original x , the perturbed data x will be used for the recommendation purpose. To achieve this, we treat numerical and categorical features separately, as these two types of data will require different perturbation strategies. Firstly, for numerical data, perturbation is performed based on a randomized encryption mechanism named piecewise mechanism (PM) [46]. Algorithm 1 shows the PM-based perturbation for each scalar numerical feature ∈ x . In PM, the original feature ∈ [−1, 1] will be transformed into a perturbed value ∈ [− , ], with defined as follows: The probability density function of the noisy output is: The following lemma establishes the theoretical guarantee of Algorithm 1. Lemma 4.1. Algorithm 1 satisfies ▷ −local differential privacy.
Proof. By Eq.(13), let , ′ ∈ [−1, 1] be any two input values and ∈ [− , ] denote the output of Algorithm 1, then we have: Thus, Algorithm 1 satisfies ▷ −LDP. □ However, the PM perturbation presented above is only designed for numerical data that is 1-dimensional. Hence, inspired by [46], we generalize Algorithm 1 to the multidimensional x containing both numerical and categorical attributes. Given x ∈ R 0 , considering it encodes ′ different features in total, we can rewrite it as is either an one-dimensional numeric or an one-hot encoding vector for a categorical feature. On this basis, we propose a comprehensive approach for perturbing such multidimensional data. The detailed perturbation process is depicted in Algorithm 2. Noticeably, we only perturb < ′ features in x . This is because that, if we straightforwardly treat each of the ′ features in x as an individual element in the dataset, then according to the composition theorem [13], the local privacy budget for each feature will shrink to ▷ ′ in order to maintain ▷ −LDP. As a consequence, this will significantly harm the utility of encrypted data. Hence, to preserve reasonable quality of each perturbed numerical or categorical feature, we propose to encrypt only a fraction of (i.e., ) features in x , ensuring a higher local privacy budget of ▷ . As shown in Algorithm 2, to prevent privacy leakage, the unselected ′ − features will be dropped by masking them with 0. Thus, to offset the recommendation accuracy loss caused by dropping these features, we follow the empirical study in [46] to determine the appropriate value of : Additionally, when perturbing each categorical feature x ( ) ∈ x , we extend the continuous sampling strategy in Algorithm 1 to a binarized version for each element/bit within the one-hot encoding x ( ) with the updated local privacy budget ▷ . As the privacy guarantee of the perturbed categorical feature x ( ) can be verified in a similar way to numerical features [48], we have omitted this part to be succinct. In this regard, our perturbation strategy for the user-centric data in recommendation can provide ▷ −LDP, as we summarize below: Lemma 4.2. Algorithm 2 satisfies ▷ −local differential privacy.
Proof. As Algorithm 2 is composed of times of ▷ −LDP operations, then based on the composition theorem [13], Algorithm 2 satisfies ▷ −LDP. □

Loss Perturbation at Optimization Stage
In most scenarios, the results generated by a predictive model (e.g., models for predicting personal credit or diseases) carry highly sensitive information about a user, and this is also the case for recommender systems, since the recommended items can be highly indicative on a user's personal interests and demographics. Though privacy can be achieved via direct perturbation on the generated results [9,30], it inevitably impedes a model's capability of learning an accurate mapping from its input to output [57], making the learned recommender unable to fully capture personalized user preferences for recommendation. Hence, in the recommendation context, we innovatively propose to perturb the ranking loss L (i.e., Eq.(11)) instead of perturbing the recommendation results in GERAI. This incurs the analysis of the privacy sensitivity Δ of L. For any function, the privacy sensitivity is the maximum L1 distance between its output values given two neighbor datasets differing in one data instance. Intuitively, the larger that Δ is, the heavier perturbation noise is needed to maintain a certain level of privacy. However, directly computing Δ from L is non-trivial due to its unbounded output range and the complex association between the input data and trainable parameters. Hence, we present a novel solution to preserving global −DP for our ranking task. Motivated by the functional mechanism (FM) [57] used for loss perturbation in regression tasks, we first derive a polynomial approximation L for L to allow for convenient privacy where L is the perturbed loss; Initialize Θ * randomly; for each ∈ U do x ← Algorithm 2; end for ∈ T do Draw a minibatch B ; L ← Eq. (17); Take a gradient step to optimize Θ * with learning rate ; end Return Θ * . sensitivity computation and make the private-preserving optimization process more generic. Then, GERAI perturbs L by injecting Laplace noise to enforce −DP. It is worth noting that, to calculate the privacy sensitivity of L, we apply a normalization step 1 to every latent predictive feature q produced in Eq.(10), which ensures every element in q is bounded by (0, 1). Using Taylor expansion, we derive L, the polynomial approximation of L: where ( ) (0) ! is the k-th derivative of L at 0. Recall that h = [ℎ 1 , ℎ 2 , ..., ℎ ] is a projection vector containing values. Let (h) = ℎ 1 1 ℎ 2 2 · · · ℎ for 1 , ..., ∈ N. Let Φ = {ℎ 1 1 ℎ 2 2 · · · ℎ | =1 = } given the degree (e.g., Φ 0 = {1}). Following [57], we truncate the Taylor series in L to retain polynomial terms with order lower than 3. Specially, only Φ 0 , Φ 1 and Φ 2 involved in L with polynomial coefficients as (0) 2! = 1 8 . Based on L, we now explore the global privacy sensitivity of the recommendation loss, denoted as Δ. Let ∈ R denote the coefficient of (h) in the polynomial. In each mini-batch training iteration, the difference of input data only influences these coefficients, so we add perturbation to L's coefficients based on the sensitivity. In the following lemma, we derive the global sensitivity Δ of L, which serves as the important scale factor in determining the noise intensity:  Proof. Given L and two training datasets D, D ′ that differ in only one instance, for ≥ 1 and q = [ 1 , 2 , ..., ] = q − q ′ , we can derive: where = ( , , ′ ) ∈ D is an arbitrary training sample and dim(·) returns the dimension of a given vector. □ Specifically, we employ FM to perturb the loss L by injecting Laplace noise 2 ( Δ | D | ) into its polynomial coefficients, and the perturbed function is denoted by L. The injected Laplace noise with standard deviation of Δ | D | has been widely proven to effectively retain −DP after perturbation [12,13,57]. Note that as Δ is the global sensitivity, it is evenly distributed to all instances in the training set D during perturbation. We showcase the full training process of GERAI with a differentially private loss in Algorithm 3. In Algorithm 3, we first compute the sensitivity Δ of loss L. In each iteration, we add perturbation to every coefficient in the polynomial approximation of the loss function. Afterwards, we launch the training session for GERAI with perturbed user feature vectors { x | ∈ U}, where we use the perturbed coefficients to obtain the perturbed loss L and optimize the parameters of the model by minimizing L. Finally, we formally prove that Algorithm 3 satisfies −DP: Lemma 4.4. Algorithm 3 maintains −differential privacy.
Proof. Assume that D and D ′ are two training datasets differing in only one instance denoted by and ′ , then we have: Then according to Definition 1, Algorithm 3 satisfies −DP. □ 2 In our paper, the mean of our Laplace distribution is 0, i.e., ( ·) = (0, ·). -Number of rated products -Number and ratio of each rating level given by a user -Ratio of positive and negative ratings: The proportions of high ratings (4 and 5) and low ratings (1 and 2) of a user.
-Entropy of ratings: It is calculated as − ∀ log , where is the proportion that a user gives the rating of .
-Median, min, max, and average of ratings -Gender: It is either male or female. In short, with our proposed dual-stage perturbation strategy for both the user data and the training loss, GERAI fully preserves user privacy with a demonstrable guarantee, while being able to achieve minimal compromise on the recommendation effectiveness compared with a non-private, GCN-based counterpart. Furthermore, GERAI can be trained via stochastic gradient descent (SGD) algorithms in an end-to-end fashion, showing its real-world practicality.

EXPERIMENTS
In this section, we conduct experiments to evaluate the performance of GERAI in terms of both privacy strength and recommendation effectiveness. Particularly, we aim to answer the following research questions (RQs): • RQ1: Can GERAI effectively protect sensitive user data from attribute inference attack? • RQ2: How does GERAI perform in top-recommendation? • RQ3: How does the key hyperparameters affect the privacypreserving property and recommendation accuracy of GERAI? • RQ:4 What is the contribution from each part of the dual-stage perturbation paradigm in GERAI? • RQ5: Can GERAI defend different types of unseen attribute inference attack models?

Dataset
Following [3], we use the publicly available ML-100K datasets [1] in our experiments. It contains 10, 000 ratings from 943 users on 1, 682 movies collected from the MovieLens website. In addition, in the collected dataset, each user is associated with three sensitive attributes, i.e., gender (Gen), age (Age) and occupation (Occ). Similar to [3], we convert the gender, age and occupation into a 2, 3 and 21-dimensional categorical feature, respectively. Table 1 provides a summary of all the features we have used.

Baseline Methods and Parameter Settings
We evaluate GERAI by comparing with the following baselines: • BPR: It is a widely used non-private learning-to-rank model for recommendation [41]. • GCN: This is the non-private, GCN-based recommendation model proposed in [55]. • Blurm: This method directly uses perturbed user-item ratings to train the recommender system [50].  • DPAE: In DPAE, Gaussian mechanism is combined in the stochastic gradient descent process of an autoencoder-based recommender so that the training phase meets the requirements of differential privacy [32]. • DPNE: It aims to develop a differentially private network embedding method based on matrix factorization, and it is the state-ofthe-art privacy preserving network embedding method for link prediction [53]. • DPMF: It uses objective perturbation with matrix factorization to ensure the final item profiles satisfy differential privacy [24]. • RAP: It is the state-of-the-art recommendation model that is designed against attribute inference attacks [3]. The key idea is to facilitate adversarial learning with an RNN-based private attribute inference attacker and a CF-based recommender.
In GERAI, we set , learning rate and batch size to 0.01, 0.005 and 64, respectively. Without special mention, we use three-layer networks for the neural components and initialized parameters to random values by using Gaussian distribution, which has 0 mean and a standard deviation of 1. The final embedding dimension is = 60 and the privacy budget is = 0.4 and ▷ = 20, while the effect of different hyperparameter values will be further discussed in Section 5.6. For all baseline methods, we use the optimal hyperparameters provided in the original papers.

Evaluation Protocols
Attribute Inference Attack Resistance. To evaluate all models' robustness against attribute inference attacks, we first build a strong adversary classifier (i.e., attacker). Specifically, we use a two-layer deep neural network model as the attacker. Suppose there are items R ( ) recommended by a fully trained recommender to user ∈ U, then the input of the attacker is formulated as ∀ ∈I ( ) ℎ ( ) + ∀ ∈R ( ) ℎ ( ) where ℎ (·) returns the one-hot encoding of a given item. The hidden dimension is set to 100, and a linear projection is used to estimate the class of the target attribute. We randomly choose 80% of the labelled users to train the attacker, and use the remainder to test the attacker's inference accuracy. Note that the attacker model is unknown to all recommenders during the training process. To quantify a model's privacy-preserving capability, we leverage a widely-used classification metric F1 score [60] to evaluate the classification performance of the attacker. Correspondingly, lower F1 scores demonstrate higher resistance to this inference attack.
Recommendation Effectiveness. For each user, we randomly pick 80% of her/his interacted items to train all recommendation models, while the rest 20% is held out for evaluation. We employ @ and @ , which are two popular metrics to judge the quality of the top-ranking list. Results on both attribute inference and recommendation are averaged over five runs. Table 2 shows the F1 scores achieved by the attribute inference attack model described in Section 5.3 on all the baselines. Lower F1 scores show higher resistance of the recommender to attribute inference attacks. Obviously, GERAI constantly outperforms all baselines with ∈ {15, 20, 25, 30}, indicating that our model is able to protect users' privacy and produce recommendations with strong privacy guarantee. Though RAP achieves slightly better results on the age attribute at = 5 and = 10, it falls behind GERAI in all other cases. As a model specifically designed for supervised learning, RAP is naturally robust against attribute inference attack. We also observe that GERAI has significantly better performance against attribute inference attack in comparison to Blurm that obfuscates user-item rating data to the recommender system. The results confirm the effectiveness of our dual-stage perturbation in private attribute protection. In addition, compared with conventional recommender systems that collaboratively model user-item interactions (i.e., BPR and GCN), models that make use of differential privacy (i.e., DPAE,  DPMF, DPNE and GERAI) show obvious superiority in resistance to attribute inference attack. However, compared with all DP-based recommender systems, GERAI achieves significantly lower 1 score for all three private attributes and thus outperform those methods in terms of obscuring users' private attribute information. The reason is that the proposed privacy mechanisms in those DP-based methods cannot have the same strength as GERAI on preventing leakage of sensitive information from recommendation results. This further validates that incorporating differential privacy may prevent directly disclosing private attributes, but these methods cannot effectively provide higher privacy levels. Furthermore, with the increasing value of , the performance of the attacker slightly decreases. One possible reason is that, more recommended products will become a natural "noise" to help reduce the risk of privacy disclosure. Finally, we observe that GCN has the weakest privacy protection results because it directly incorporates the node features with sensitive information. Note that compared with GCN, GERAI achieves an average relative improvement of 11%, 14.4% and 6.75% respectively on age, gender and occupation, which implies that DP can ensure that the published recommendations of GERAI can avoid breaching users' privacy.

Recommendation Effectiveness (RQ2)
We summarize all models' performance on personalized recommendation with Table 3. Note that higher @ and @ values imply higher recommendation quality. Firstly, GERAI outperforms all privacy-preserving baselines consistently in terms of both @ and @ . Particularly, the improvement of GERAI with = 5 demonstrate that our model can accurately rank the ground truth movies at the top-5 positions. In addition, compared with RAP, GERAI yields recommendation results that are closer to the state-of-the-art GCN. Thanks to the dual-stage perturbation setting where two sets of privacy budgets are used, a relatively higher privacy for user feature perturbation does not significantly impede the recommendation accuracy, and is sufficient for highlevel attribute protection. Furthermore, the gap between the ranking accuracy drops with the increasing value of . Finally, GCN achieves the best performance among all methods except when = 30, which showcases the intrinsic strength of GCN-based recommenders. Meanwhile, Blurm has the worst performance among all methods as the way it adds noise to the user-item interaction data is harmful for the recommendation quality.

Accuracy and Privacy (RQ3)
We answer RQ3 by investigating the performance fluctuations of GERAI with varied global and local privacy budgets , ▷ and embedding dimension . We vary the value of one hyperparameter while keeping the other unchanged, and record the new recommendation and attribute inference results achieved. Figure 2 plots the results with different parameter settings.
Impact of Global Privacy Budget for Loss Perturbation. The value of the privacy budget is examined in {0.1, 0.2, 0.4, 0.8, 1.6, 3.2}. In general, our GERAI outperforms RAP in terms of recommendation accuracy, and the performance improvement tends to become less significant when becomes quite small. Since a smaller requires a larger amount of noise to be injected to the objective function, it negatively influences the recommendation results. The results further confirms the effectiveness of GCNs-based recommendation component in our model, which helps GERAI preserve recommendation quality in practice. Furthermore, though the attack results illustrate that a relatively small (large noise) can obtain better performance on privacy protection within our expectation, it also degraded recommendation results correspondingly. Compared with RAP, the results imply that, by choosing a proper value of (0.4 in our case), our GERAI can achieve a good trade-off between privacy protection and recommendation accuracy.
Impact of Local Privacy Budget ▷ for User Feature Perturbation. We study the impact of the privacy budget on input features with ▷ ∈ {0.5, 5, 10, 20}. It is worth mentioning that we seek a relatively higher value of ▷ to maintain moderate utility of user features. From Figure 2, we can draw the observation that though reducing the value of privacy budget ▷ in the input features may help the model yield better performance against attribute inference attack, GERAI generally achieves a significant drop on recommendation performance with a smaller ▷ . Particularly, when ▷ = 0.5, the recommendation results show that GERAI cannot capture users' actual preferences. This is because the feature vector x determines the number of non-zero elements in base embedding of our model, which can cause significant information loss when it is small. As the recommendation is also highly accurate when ▷ = 10, the attribute inference performance achieved by the attacker is occasionally comparable to setting ▷ = 20. Overall, setting ▷ to 20 is sufficient for preventing privacy leakage, while helping GERAI to achieve optimal recommendation results.
Impact of Dimension . As suggested by Eq.(18), the dimension controls the privacy sensitivity Δ and our model's expressiveness of the network structure. We vary the dimension in {20, 40, 60, 80, 100} and the corresponding noise parameters in Laplace distribution are {0.00375, 0.01375, 0.03, 0.05, 0.08}. Obviously, the recommendation accuracy of GERAI benefits from a relatively larger dimension , but the privacy protection performance is not always lower with a large . The reason is that the value of the dimension is directly associated with our model's expressiveness, which means that a relatively larger can improve the recommendation results, providing better inputs to the attacker model as well. Furthermore, as shown in Figure 2, the best privacy protection performance is commonly observed with = 60.

Importance of Privacy Mechanism (RQ4)
To better understand the performance gain from the major components proposed in GERAI, we perform ablation analysis on different degraded versions of GERAI. Each variant removes one privacy mechanism from the dual-stage perturbation paradigm. Table 4 summarizes the outcomes in two tasks in terms of @5, @5 and F1 score. For benchmarking, we also demonstrate the results from the full version of GERAI and the non-private GCN.
Removing perturbation at input stage (GERAI-NL). The GERAI-NL only enforces -differential privacy by perturbing the objective function in Eq. (17). We remove the privacy mechanism in users' features by sending raw features X directly into the recommendation component. After that, a slight performance decrease in the recommendation accuracy appeared, while achieving better performance against attribute inference attack. The results confirm that the functional mechanism in our model can help a GCN-based recommender satisfy privacy guarantee and yield comparable recommendation accuracy. In addition, GERAI significantly outperform GERAI-NL against attribute inference attack. Apparently, the raw user features are not properly perturbed in GERAI-NL, leading to a high potential risk in privacy leakage.
Removing perturbation at optimization stage (GERAI-NF). We remove the privacy mechanism in objective function by setting = 0. As the users' features are perturbed against information leaks, GERAI-NF achieves a significant performance improvement in the privacy protection, compared with the pure GCN. In addition, the slight performance difference between GERAI and GERAI-NF in two tasks could be attributed to the perturbation strategy in objective function. It further verifies that the joint effect of perturbation strategies in objective function and input features are beneficial for both recommendation and privacy protection purposes.

Robustness against Different Attribute Inference Attackers (RQ5)
In real-life scenarios, the models used by attribute inference attacker are usually unknown and unpredictable, so hereby we investigate how GERAI and other baseline methods perform in the presence of different types of attack models, namely Decision Tree (DT), Naive Bayesian (NB), KNN and Gaussian Process (GP), that are widely adopted classification methods. In this study, we use the top-5 recommendation generated by corresponding recommender methods for all attackers as introduced in Section 5.3. Table 5 shows the attribute inference accuracy of each attacker. The first observation is that our proposed GERAI outperforms all the comparison methods in most scenarios. Though DPAE achieves slightly better results in several cases, its recommendation accuracy is non-comparable to GERAI. This further validates the challenge of incorporating privacy protection mechanism for personalized recommendation. Another observation is that there is a noticeable performance drop of RAP facing non-DNN attacker models. As RAP is trained to defend a specific DNN-based inference model, RAP is more effective when attacker is also DNN-based as shown in Table 2. However, RAP underperforms when facing the other five commonly used inference models, showing that GERAI can more effectively resist attribute inference attacks and protect users' privacy without any assumption on the type of attacker models.

RELATED WORK
Attribute Inference Attacks. The target of attribute inference attack is inferring users' private attribute information from their publicly available information (e.g. recommendations). Three main branches of attribute inference attack approaches are often distinguished: friend-based, behavior-based and hybrid approaches. Friend-based approaches infer the target user's attribute in accordance with the target's friends' information [19,22,31]. He et al [22] first constructed a Bayesian network to model the causal relations among people in social networks, which is used to obtain the probability that the user has a specific attribute. Behavior-based approaches achieve this purpose via users' behavioral information such as movie-rating behavior [50] and Facebook likes [29]. The third type of works exploits both friend and behavioral information [17,18,23]. For example, [19] creates a social-behaviorattribute network to infer attributes. Another work [23] models structural and behavioral information from users who do not have the attribute in the training process as a pairwise Markov Random Field. Privacy and Recommender System. With the growth of online platforms (e.g. Amazon), recommender systems play a pivotal role in promoting sales and enhancing user experience. The recommendations, however, may pose a severe threat to user privacy such as political inclinations via attribute inference attack. Hence, it is of paramount importance for system designers to construct a recommender system that can generate accurate recommendations and guarantee the privacy of users. Current researches that address vulnerability to privacy attacks often rely on providing encryption schemes [6,27] and differential privacy [25]. Encryption-based methods enhance privacy of the conventional recommender systems with advanced encryption techniques such as homomorphic encryption [8,27]. However, these methods are considered computation expensive as a third-party crypto-service provider is required. DP-based recommender systems can provide a strong and mathematically rigorous privacy guarantee [4,33,34]. Works in this area aim to ensure that the recommender systems are not sensitive to any particular record and thus prevent adversaries from inferring a target user's ratings. [38] proposes a perturbation method that adds or removes items and ratings to minimize privacy risk. Similarly, RAPPOR [15] is proposed to perturb the user's data before sending them to the server by using the randomized response. More recently, graph embedding techniques have been opening up more chances to improve the efficiency and scalability of the existing recommender systems [7,55]. As the core of GCN is a graph embedding algorithm, our work is also quite related to another area: privacy preservation on graph embedding. Hua et al. [24] and Shin et al. [43] proposed gradient perturbation algorithms for differentially private matrix factorization to protect users' ratings and profiles. Another work enforces differential privacy to construct private covariance matrices to be further used by recommender [12]. Liu et al. [32] proposed DPAE that leverages the privacy problem in recommendation with the Autoencoders. Gaussian noise is added in the process of gradient descent. However, the existing privacypreserving works in recommendation systems focus on protecting users against the membership attacks in which an adversary tries to infer a targeted user's actual ratings and deduce if the target is in the database, which is not fulfilled in our scenario. These limitations motivated us to propose GERAI that is able to counter private attribute inference attacks in the personalized recommendation system.

CONCLUSION
In this paper, we propose a GCN-based recommender system that guards users against attribute inference attacks while maintaining utility, named GERAI. GERAI firstly masks users' features including sensitive information, and then incorporates differential privacy into the GCN, which effectively bridges user preferences and features for generating secure recommendations such that a malicious attacker cannot infer their private attribute from users' interaction history and recommendations. The experimental results evidence that GERAI can yield superior performance on both recommendation and attribute protection tasks.

ACKNOWLEDGMENTS
The work has been supported by Australian Research Council (Grant No.DP190101985 and DP170103954).